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Abstract 

Military  leaders  stress  the  need  for  realistic 
exercises  in  training  military  forces.  However,  few  people 
have  addressed  how  exercise  realism  impacts  unit  readiness. 
The  purpose  of  this  research  was  to  determine  the 
relationship  between  exercise  realism  and  exercise 
effectiveness.  The  Delphi  method  and  expert  opinion  was 
used  to  examine  several  issues  involving  realism  in  Air 
Force  exercises.  The  current  level  of  exercise  realism  was 
examined,  and  the  impact  of  increasing  realism  in  Air  Force 
exercises  was  explored.  Additionally,  several  factors  that 
affect  realism  were  identified.  These  factors  were  analyzed 
to  determine  which  factors  contributed  most  significantly  to 
exercise  effectiveness  and  unit  combat  readiness. 

The  results  indicated  that  realism  is  related  to  the 
effectiveness  of  an  exercise  in  preparing  forces  for  combat; 
several  factors  that  affected  realism  also  significantly 
impacted  unit  combat  readiness.  Understanding  the 
relationship  between  exercise  realism  and  unit  readiness 
will  help  Air  Force  planners  conduct  more  effective  and 
efficient  exercises. 
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THE  RELATIONSHIP  BETWEEN  REALISM 


IN  AIR  FORCE  EXERCISES  AND  COMBAT  READINESS 
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general  ..Issue 

The  mission  of  the  United  States  Air  Force  (USAF)  is  to 
defend  the  United  States  through  control  and  exploitation  of 
air  and  space  power  (25:5).  This  mission  statement, 
delivered  recently  by  USAF  Chief  of  Staff  General  Merrill  A. 
McPeak,  emphasizes  the  broad  and  complex  responsibilities  of 
the  USAF.  To  meet  this  global  responsibility,  Air  Force 
members  must  be  adequately  prepared  to  perform  their  unit's 
wartime  mission. 

Air  Force  personnel  must  receive  realistic  training 

that  prepares  them  for  combat.  Air  Force  Manual  ( AFM)  1-1, 

Volume  1,  stresses  the  need  for  realistic  training: 

Training  should  be  as  realistic  as  possible.  Aerospace 
forces  must  train  as  they  plan  to  fight.  Exercises 
must  replicate  to  the  extent  possible  the  chaos, 
stress,  intensity,  tempo,  unpredictability,  and 
violence  of  war.  Further,  exercises  must  include 
"free-play"  scenarios  that  emphasize  innovative  problem 
solving,  rapidly  changing  situations,  and  degraded 
capabilities.  (7:18) 

AFM  1-1,  Volume  2,  continues  by  stating  that  frequent  and 
realistic  exercises  provide  one  of  the  most  effective  forms 
of  training.  Further,  the  more  "true-to-lif e  and  stressful 
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exercises  are  in  peacetime,  the  better  prepared  the  force 

will  be  to  survive  and  win  in  combat"  (8:241). 

Many  people  advocate  realistic  training  and  understand 

its  importance  in  preparing  forces  for  combat.  Lieutenant 

General  Michael  Nelson,  Deputy  Chief  of  Staff,  Operations 

and  Plans,  Headquarters  USAF,  wrote  the  following  in  an 

essay  about  the  United  States'  Persian  Gulf  War  with  Iraq: 

Our  forces  were  ready  for  the  conflict  in  January  and 
February  1991  because  of  the  dollars  we  spent  on 
training,  readiness,  and  parts  in  the  1980s.  We  were 
ready  in  January  because  we  made  a  commitment  in  the 
eighties  to  realistic  training.  (28:119) 

Lieutenant  General  Charles  A.  Horner,  commander  of  U.S.  Air 

Forces  during  the  Gulf  War,  also  emphasized  the  important 

role  that  exercises  played  in  the  success  of  that  war: 

Annual  and  biannual  exercises  such  as  GALLANT  KNIGHT, 
GALLANT  EAGLE,  BRIGHT  STAR,  QUICK  FORCE,  BLUE  FLAG,  and 
RED  FLAG  paved  the  way  to  realistic  and  pragmatic 
expectations.  As  the  years  passed,  we  honed  our 
ability  to  conduct  air  operations  and  fight  in  the 
desert  and  grew  in  our  understanding  of  the  uniqueness 
of  our  area  of  responsibility.  Deployments  in  the 
desert  gave  us  an  understanding  of  the  effects  of  heat, 
sand  and  dust  on  our  personnel  and  equipment  and 
allowed  us  to  make  adequate  preparations  to  overcome 
the  elements.  (18:2) 

The  U.S.  Central  Command  (CENTCOM)  Exercise  Division 
Director  during  Operation  Desert  Shield/Storro  (ODS),  Colonel 
Jack  A.  Klimp,  wrote  that  prior  to  Desert  Shield,  exercises 
were  conducted  at  various  times  with  nine  of  the  nations  in 
the  area  of  responsibility  (AOR).  The  benefits  of  these 
experiences  were  manifested  during  ODS  once  the  forward 
headquarters  element  had  deployed.  Klimp  notes  that  the 
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exercises  created  a  large  contingent  of  staff  officers  who 
were  experienced  in  the  rigors  of  Middle  Eastern  desert 
warfare.  These  officers  had  travelled  extensively 
throughout  the  AOR  and  were  familiar  with  the  region's 
environment,  climate,  and  infrastructure  and  how  each 
affected  operations.  They  were  also  experienced  in  dealing 
with  AOR  military  forces,  some  to  the  point  that  they  had 
actually  fostered  relationships  with  host-nation  officials. 
Perhaps  more  critically,  they  were  experienced  in 
joint/combined  planning  and  execution  (18:12). 

General  Carl  von  Clausewitz  is  another  military  leader 
who  recognized  the  need  for  realistic  training.  In  his 
book.  On  War .  he  writes  that  "practice  and  experience  teach 
a  commander  what  is  possible  and  what  is  not"  (4:20).  He 
later  writes  that  the  only  " lubricant"  which  reduces  war's 
friction  is  combat  experience.  However,  although  he  asserts 
that  exercises  are  a  poor  substitute  for  the  real  thing,  he 
concedes  that  the  forces  that  use  this  training  tool  will  be 
far  better  prepared  than  forces  who  practice  routine, 
mechanical  drills  (4:121-2).  Clearly,  realistic  training  is 
important.  For  military  forces,  ignoring  realism  carries 
the  risk  of  being  unprepared  for  war. 


The  importance  of  combat  readiness  has  been  recognized 
by  military  thinkers  since  antiquity.  Almost  5,000  years 


1-3 


ago,  the  Chinese  military  philosopher  Sun  Tzu  wrote: 

It  is  a  doctrine  of  war  not  to  assume  the  enemy  will 
not  come,  but  rather  to  rely  on  one's  readiness  to  meet 
him;  not  to  presume  that  he  will  not  attack,  but  rather 
to  make  one's  self  invincible.  (31:114) 

Exercises  have  played  a  crucial  role  in  attaining  and 

building  the  competence  necessary  to  conduct  successful 

military  operations.  The  militaries  of  Germany  and  Japan 

used  war  games  to  prepare  for  their  aggressive  assaults  that 

launched  World  War  II  (32:206).  The  impressive  victory  at 

Pearl  Harbor  and  the  devastating  blitzkrieg  of  Europe  are 

good  examples  of  battlefield  success  obtained  through 

effective  training.  Operation  OVERLORD,  the  D-Day  Normandy 

landing  that  spearheaded  the  Allied  drive  to  Berlin,  was 

practiced  for  nearly  two  years  before  execution  (32:207). 

While  the  importance  of  realism  in  exercises  is 

recognized  by  military  leaders,  increases  in  exercise 

realism  carry  a  corresponding  increase  in  the  cost  of 

conducting  exercises.  AFM  1-1,  Volume  2  states  that 

"exercises  consume  money,  time,  and  other  limited  resources" 

(8:241).  Exercises  are  important,  out  they  must  be 

conducted  in  a  manner  that  maximizes  combat  readiness  while 

managing  costs. 

The  Office  of  the  Secretary  of  Defense  (OSD)  recently 
proposed  reductions  in  funding  for  real-world  deployments 
and  field  training  exercises  (FTXs).  OSD  anticipates  that 
an  expansion  in  the  use  of  command  post  exercises  (CPXs), 
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and  simulations  will  make  up 


computer  generated  war  ga'ies, 
the  shortfall.  Recent  reauctions  in  military  budgets  have 
placed  greater  emphasis  on  reducing  costs,  including  those 
associated  with  training.  Reducing  the  level  of  realism  in 
exercises  will  undoubtedly  reduce  costs;  however,  these  cost 
reductions  may  be  at  the  expense  of  force  readiness 
(16:32;  17:20). 

An  examination  of  U.S.  military  history  clearly 
indicates  the  tendency  of  the  nation  to  drastically  reduce 
military  expenditures  after  every  major  conflict  since  the 
American  Revolution.  History  also  demonstrates  that  this 
invariably  reduces  the  military's  subsequent  state  of  combat 
readiness — sometimes  with  disastrous  consequences 
(1:22,26,47).  Recent  events  demonstrate  that  the  end  of  the 
Cold  War  may  be  no  exception. 

Increasing  public  pressure  to  curtail  federal  spending 
has  focused  attention  on  reducing  military  budgets.  The 
challenge  for  today's  military  establishment  is  to  absorb 
these  reductions  while  minimizing  the  negative  impact  on 
readiness.  To  accomplish  reductions  in  exercise  spending 
while  maintaining  combat  readiness,  one  roust  understand 
exercise  realism  and  combat  effectiveness  and  how  these  two 
variables  interact. 

A  tradeoff  exists  between  the  level  of  realism  in  an 
exercise  and  the  effectiveness  of  the  exercise  in  preparing 
forces  for  combat;  however,  the  extent,  or  strength,  of  this 
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relationship  has  not  been  explored.  Military  experts  agree 
that  training  exercises  should  be  realistic  (7:18;  28:119). 
The  existing  research,  however,  does  not  address  the 
perceptions  of  current  levels  of  realism  in  exercises  and 
how  this  realism  relates  to  combat  readiness.  Must  all 
exercises  be  as  realistic  as  possible  to  be  effective  and 
improve  combat  readiness?  What  elements  of  realism  are 
crucial  tc  the  effectiveness  of  an  exercise,  and  what 
elements  contribute  little  or  nothing?  These  relationships 
are  the  focus  of  this  research. 

Problem  Statement 

The  purpose  of  this  research  is  to  determine  the 
relationship  between  the  level  of  realism  in  Air  Force 
exercises  and  exercise  effectiveness.  This  research 
examines  some  of  the  factors  that  contribute  to  exercise 
realism  and  how  these  factors  explain  the  effectiveness  of 
exercises  in  terms  of  improved  combat  readiness. 

Investigative  Questions 

fpecific  questions  to  research  include  the  following: 

1.  What  are  the  perceptions  of  Air  Force  members 
concerning  the  level  of  realism  in  exercises? 

2.  What  are  Air  Force  members'  perceptions  of  the 
amount  of  realism  and  its  impact  on  the  effectiveness  of  an 
exercise? 
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3.  Are  there  significant  differences  in  the  level  of 
realism  utilized  in  exercises  among  major  commands  and 
management  levels? 

4.  What  are  some  factors  that  contribute  to  exercise 
realism? 

5.  How  do  these  factors  relate  to  exercise 
effectiveness  and  combat  readiness? 

6.  What  are  some  advantages  associated  with 
increasing  realism  in  exercises? 

7.  What  are  some  disadvantages  associated  with 
increasing  realism  in  exercises? 

Scope 

The  research  objective  is  to  learn  as  much  as  possible 
about  the  perceptions  of  realism  in  Air  Force  training 
exercises.  The  scope  is  intentionally  broad  and  does  not 
focus  on  a  particular  type  of  exercise  or  element  of  the 
exercise  system.  Additionally,  the  research  includes  data 
from  several  major  commands,  career  fields,  and  management 
levels  to  gain  an  overall  perspective  of  Air  Force  exercise 
realism. 

Definition  of  Terms 

Several  terras  must  be  defined  to  prevent  confusion 
caused  by  different  meanings.  These  key  terms  are 
operationally  defined  in  the  manner  they  are  used  in  this 
research. 


1-7 


Combat  Readiness.  The  condition  of  a  unit  being  ready,  in 
terms  of  the  concept  or  plan  of  a  particular  operation,  to 
carry  out  the  operations  required  by  the  mission  of  that 
unit  (14:360).  To  avoid  redundancy  this  term  is  used 
interchangeably  with  unit  readiness  and  combat 
effectiveness. 

Exercise  Effectiveness.  The  ability  of  an  exercise  to 
improve  the  combat  readiness  of  military  personnel  or  units. 
Military.  Exercise.  Any  activity  involving  the  operation  of 
actual  military  forces  in  an  artificial  or  simulated  hostile 
environment  (11:9). 

Readiness.  s'The  state  of  preparedness  of  an  individual, 
force,  or  organization  for  carrying  out  an  operation, 
mission,  task,  or  the  like”  (14:427). 

Realism/Reality .  The  degree  to  which  a  scenario  or  exercise 
reflects  the  actual  tasks  to  be  accomplished,  conditions, 
and  environment  that  a  unit  would  likely  be  confronted  with 
in  a  contingency  or  combat  situation.  Reality  consists  of 
numerous  physical  and  psychological  elements  which  include 
factors  such  as  time,  task,  place,  environment,  equipment, 
knowledge,  infrastructure,  and  human  stress  and  emotion. 
Scenario .  A  scripted  play  of  events,  usually  in  the  form  of 
an  artificial  threat  or  problem  which  may  jeopardize  the 
accomplishment  of  the  unit's  mission. 
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Summary 


The  purpose  of  this  research  is  to  explore  the 
relationship  between  exercise  realism  and  exercise 
effectiveness.  The  specific  investigative  questions  will  be 
answered  in  the  following  chapters.  Chapter  II  is  a  review 
of  the  existing  literature  relevant  to  Air  Force  training 
exercises,  realism,  and  combat  readiness.  Chapter  III 
discusses  the  specific  methodology  used  for  data  collection, 
and  Chapter  IV  presents  the  analysis  of  the  data.  Chapter  V 
reports  the  results  of  the  analysis  and  offers 
recommendations  as  a  result  of  the  research. 
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Most  military  experts  agree  that  training  should  be 
realistic.  However,  little  research  has  been  done  that 


explains  how  realism  in  an  exercise  relates  to  unit  combat 
readiness.  First,  this  chapter  examines  the  two  general 
categories  of  military  exercises  and  explores  some 
limitations  of  the  exercises  that  can  reduce  their 
effectiveness.  Next,  this  review  examines  the  existing 
literature  on  exercise  realism  and  unit  combat  readiness. 
The  relationship  between  realism  and  readiness  is  explored 
and  obstacles  that  may  reduce  exercise  realism  are 
presented. 


Types  ot  E&erciggs 

An  exercise  is  nany  activity  involving  the  operation  of 

actual  military  forces  in  a  simulated  hostile  environment" 

(5:9).  The  Conduct  of  the  Persian  Gulf  Conflict.  Final 

Report  to  Congress  clearly  states  the  purpose  of  exercises: 

Large  scale  exercises  provide  an  opportunity  to 
synchronize,  maneuver,  and  support  forces  in  realistic, 
stressful  situations.  Short  of  combat,  exercises  are 
the  best  method  to  determine  training  and  readiness 
strengths  and  weaknesses.  (18:46) 

The  Air  Force  conducts  numerous  exercises,  each  with 
specific  training  objectives  and  rules  of  engagement; 
however,  ail  military  exercises  can  be  grouped  into  two  main 


categories:  field  training  exercises  (FTXs)  and  command  post 
exercises  (CPXs).  Jn  a  FTX,  forces,  equipment,  personnel, 
and  aircraft  actually  move  to  some  location  where  the 
exercise  scenario  is  conducted  under  simulated  wartime 
conditions.  FTXs  provide  realistic  training  for  United 
States  and  allied  combat  forces  and  help  foster  greater 
cooperation  (11:6).  An  additional  benefit  of  the  FTX,  when 
conducted  in  a  foreign  nation,  is  that  it  enables  the  United 
States  to  establish  a  presence  in  that  nation  without  the 
expense  of  building  and  maintaining  a  full-scale  operating 
base  (8:33). 

Unlike  the  FTX,  CPXs  simulate  the  movement  of  forces 
and  equipment  and  emphasize  communication  and  coordination 
capabilities  between  various  headquarters  and  subordinate 
commands.  Large  scale  CPXs  are  usually  driven  by  computer 
databases  and  are  much  less  expensive  to  conduct  than  FTXs. 
CPXs  are  growing  in  importance  because  of  their  lower  cost 
and  the  increasing  technology  available  to  produce  more 
complex  environments  (20:26). 

Limitations  of  FTXs 

There  are  several  peacetime  limitations  that  reduce  the 
scope  and  effectiveness  of  FTXs.  These  limitations  are  due 
to  three  main  constraints:  fiscal,  political,  and 
environmental.  Field  exercises  provide  good  hands-on 
training  opportunities,  but  the  exercises  are  expensive 
(13:1).  The  movement  of  forces  and  equipment  may  account 
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for  up  to  eighty  percent  of  the  total  exercise  budget 
(27:5).  Because  projections  for  the  funding  of  exercises 
across  Air  Force  major  commands  are  only  seventy  to  eighty 
percent  of  the  amount  requested,  alternative  sources  of 
training  are  beginning  to  play  a  more  significant  role  in 
preparing  forces  for  war  (18:33;  20:25).  These  budget 
constraints  prevent  exercise  planners  and  commanders  from 
maximizing  the  amount  of  realism  in  exercises. 

Political  constraints  can  also  effect  FTXs  and  reduce 
their  level  of  realism.  Many  exercises  require  the 
involvement  of  host  nations  to  approve  exercise  durations, 
provide  support  facilities,  and  select  operational  areas. 
Although  United  States  and  allied  forces  benefit  by 
interacting  and  learning  about  each  others  capabilities, 
current  political  sensitivities  and  international  events  can 
limit  an  exercise's  scope  and  realism.  For  example, 
diplomatic  landing  clearances,  overflight  restrictions,  and 
host  nation  holidays  can  limit  the  ability  of  exercise 
planners  to  test  military  capabilities  (27:6). 

Finally,  environmental  issues  such  as  noise  abatement, 
agricultural  damage,  and  safety  regulations  can  limit  the 
scheduling,  location,  and  realism  of  exercises.  For 
example,  personnel  safety  is  more  critical  during  peacetime 
maneuvers  than  during  war  when  missions  typically  require 
troops  to  put  their  lives  at  risk  (27:4-6). 
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Limitations  of  CPXs 


Command  post  exercises  simulate  the  movement  of  forces 
and  equipment  and  are  less  costly  than  field  exercises. 
However,  CPXs  have  unique  limitations  that  can  reduce  their 
realism.  These  exercises  rely  on  scenario  scripts  driven  by 
computer  databases.  The  realism  achieved  during  exercise 
play  is  dependent  on  the  quality  of  the  database  and  the 
capabilities  of  the  exercise  control  staff  (12:8;  16:1). 

Major  John  L.  Krueger,  a  commander  and  exercise 
coordinator  in  the  Exercise  and  Simulations  Division  at  Fort 
Riley,  Kentucky,  highlights  the  importance  of  the  exercise 
control  staff: 

The  personnel  who  staff  the  simulation  center,  whether 
civilian  contractor,  government  service  civilians  or 
military,  must  be  technically  competent,  highly 
motivated  and  aggressively  proactive  to  make  the  system 
operate  to  its  fullest  potential.  A  knowledge  of 
current  weapon  systems,  tactics  and  enemy  doctrine, 
combined  with  a  thorough  knowledge  of  what  the 
simulation  can  replicate,  is  necessary  for  at  least  one 
member  of  the  staff.  (20:25) 

Another  limitation  of  the  simulation  driven  exercise  is 
what  Krueger  described  as  exercise  ’'gamesmanship"  by 
commanders.  These  individuals  often  overemphasize  the 
tactical  and  strategic  play  of  such  systems;  areas  the 
simulation  is  often  not  designed  to  realistically  portray. 

In  their  effort  to  "win,"  commanders  often  divide  their 
forces  in  ways  that  they  would  never  attempt  in  the  field. 
This  attitude  often  results  in  placing  the  main  purpose  of 
simulations,  to  practice  reporting  procedures  and  test 
communication  links,  in  a  secondary  role  (20:20-4). 
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Another  aspect  of  CPXs  that  reduces  realism  is  the 
level  of  play.  CPXs  usually  revolve  around  command  and 
control  procedures  at  the  Joint  Chiefs  of  Staff  (JCS)  and 
Unified  Command  level.  The  information  flow  typically  stops 
at  the  major  commands  and  omits  participation  by  unit  level 
commanders.  In  "Limitations  of  JCS  Exercises,"  Major  Gregg 
Perry  states: 

I  believe  we  give  up  realism  by  not  completing  the  flow 
of  information  to  the  unit  level  command  and  control 
agencies.  However,  exercise  planning  at  the  high(er) 
levels  of  government  takes  a  lot  of  time,  and  until 
many  tof  level  decisions  are  made,  there  is  little 
meaningful  involvement  available  at  the  lower  levels. 
Furthermore,  real  world  commitments  may  keep  the  same 
high  level  players  from  participating  in  these 
exercises,  and  we  lose  more  realism.  (27:9) 


Realism 

Several  authors  have  addressed  exercise  realism  in 
their  research.  Hagel  examined  the  methods  used  in 
conducting  Air  Force  CPXs  and  the  difficulties  in  achieving 
realism.  He  found  several  factors  that  made  realism  in  CPXs 
difficult  to  achieve.  Among  the  factors  limiting  exercise 
realism  were  the  following:  too  little  time  to  conduct  the 
exercise,  lack  of  enthusiasm  among  exercise  players,  lack  of 
sufficient  funding,  improper  planning,  inadequate  feedback, 
and  the  failure  of  senior  leadership  to  play  their  proper 
exercise  roles.  Through  interviews  of  thirty-three  people 
familiar  with  CPXs,  Hagel  concluded  that  "there  is  not 
enough  logistics  realism  in  the  exercise  program,  but  there 
is  some  worthwhile  training  coming  from  them"  (11:3). 
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In  his  thesis,  Hall  addressed  the  need  for  increasing 
realism  within  B-52G  CPXs.  While  developing  a  logistics 
database  to  increase  realism  in  these  CPXs,  he  determined 
that: 

Increasing  amounts  of  artificiality  have  crept  into 
many  exercises,  especially  in  the  area  of  logistics. 

As  a  result,  even  though  the  exercises  are  considered 
successful,  they  may  contribute  little  to  the  goal  of 
improving  combat  effectiveness.  (12:2-2) 

Thus,  although  many  exercises  were  completed  with  positive 

outcomes,  their  lack  of  realism  may  lead  to  different 

results  in  actual  combat. 

In  1972,  Kestler  and  Boren  measured  the  attitudes  of  unit 
managers  toward  the  Air  Force  Inspection  and  Readiness  Evaluation 
Program  at  two  now  defunct  major  commands,  the  Aerospace  Defense 
Command  (ADC)  and  the  Strategic  Air  Command  (SAC).  Among  the 
attitudinal  factors  of  interest  were  responses  from  unit  managers 
concerning  the  relevancy  of  exercise  scenarios  used  under  the 
program,  the  ability  of  the  program  to  point  out  deficiencies 
critical  to  mission  accomplishment,  and  the  overall  effectiveness 
of  the  inspection  program.  Kestler  and  Boren  concluded  that  the 
overall  program  was  valuable  but  noted  some  weakness  in  the  areas 
of  relevancy  and  criticality,  particularly  where  administrative 
matters  were  concerned.  These  findings  suggest  a  lack  of  realistic 
wartime  requirements  within  the  scenarios  used  in  the  readiness 
evaluation  program  (17:53-7). 


2-6 


Krueger  noted  that  while  CPX  simulations  are  being 
increasingly  relied  upon  as  training  tools,  they  are 
commonly  misused.  Thus,  personnel  who  participate  in  CPXs 
often  receive  minimal  benefit  from  their  use.  He  notes  that 
while  simulations  are  designed  primarily  to  emphasize 
communications,  unrealistic  tactical  play  on  the  part  of 
participants  often  reduces  the  value  of  this  type  of 
training.  Poorly  qualified  staffs  within  simulation  control 
centers  can  also  limit  the  quality  of  the  exercise  (20:25). 

While  this  literature  sampling  on  realism  is  relatively 
small,  the  emphasis  on  the  importance  of  realistic  training 
can  not  be  ignored.  The  factors  that  constitute  realism 
remain  largely  unaddressed  by  the  literature.  Hagel  vaguely 
suggests  elements  of  realism  in  his  discussion  of  obstacles 
that  reduce  exercise  realism.  Of  these  obstacles,  the  time 
available  to  conduct  exercises  and  the  attitudes  of 
participants  appear  to  be  the  most  significant.  Kestler  and 
Boren  suggest  a  third  component  of  realism,  the  task 
element.  An  effective  exercise  or  evaluation  must  include  a 
scenario  that  requires  participants  to  perform  tasks  that 
they  would  be  expected  to  perform  in  combat  (17:30). 

Combat  Readiness 

In  his  article,  "Leadership--Analysis  and  comment," 
Lieutenant  Colonel  Herbert  F.  Harback  assesses  readiness: 
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When  is  the  probability  the  highest,  the  risk  the 
greatest  that  armed  conflict  will  start  (sic)?  It  is 
when  the  bad  guy  believes  that  he  can  beat  the  good 
guy — when,  in  the  mind  - r  the  aggressor  it  is  felt  that 
the  other  person,  nation,  Army,  or  soldiers  do  not  look 
like  they  are  ready.  History  (sic)  tells  us  that  peace 
comes  through  strong  deterrence.  (13:22) 

The  literature  contains  many  sources  which  address 

strategies  for  determining  the  effectiveness  of  an 

individual  soldier  or  coiubat  unit.  The  morale  of  the 

individual  or  unit  and  the  degree  of  training  each  received 

are  universal  themes  of  these  presentations. 

In  his  study  of  the  combat  effectiveness  of  weapon 

systems,  Koraan  writes  that  the  primary  elements  in 

determining  a  unit's  combat  effectiveness  are:  1)  the 

quality  of  the  commanders,  2)  the  will-power,  morale  and 

discipline  of  the  unit,  3)  the  type,  qurlity,  and  number  of 

arms  the  unit  possesses,  4)  the  physical  conditions  of  the 

soldiers,  and  5)  the  training  level  and  combat  experience  of 

the  unit  (19:1).  Homan  believed  that  these  factors  fall 

under  two  broader  categories:  those  that  are  stri  tly 

quantifiable  and  those  that  can  best  be  described  as 

concepts. 

In  their  contract  report,  "Identification  of  Observable 
Factors  Related  to  Success  in  Combat  or  Simulated  Combat," 
Edgarton  and  Graham  explored  the  differences  between 
successful  and  unsuccessful  infantrymen  in  combat  and 
simulated  combat  situations.  Although  developed  as  a 
psychological  study  of  the  individual  behavior  traits  which 
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cause  a  soldier  to  be  either  effective  or  ineffective,  this 
research  yields  interesting  insights  (9:2). 

Through  their  literature  review,  Edgarton  and  Graham 
discovered  that  "buddy”  ranking  provided  more  accurate 
predictors  of  combat  success  of  individuals  than  did  ratings 
by  the  individuals'  superiors.  Using  questionnaire  data 
from  veterans  of  World  War  II  and  the  Korean  War  and 
soldiers  undergoing  basic  and  advanced  infantry  training, 
the  researchers  found  that  those  soldiers  rated  effective  by 
their  peers  scored  hignest  in  the  following  four  areas: 
positive  reactions  under  fire,  leadership  qualities,  social 
responses  in  terms  of  cooperation,  loyalty,  and  esprit  de 
corp,  and  intelligence  in  terms  of  knowledge,  recognition, 
and  tactical  ingenuity  (9:9-36).  If  it  can  be  demonstrated 
that  realistic  training  develops  or  strengthens  these 
qualities,  then  realism  can  be  directly  linked  to  combat 
effectiveness.  This  relationship  between  realistic  training 
and  the  development  of  these  four  areas  is  indirectly 
supported  by  Edgarton  and  Graham.  They  show  that  the  longer 
that  individuals  ar"'  observed  under  fire,  or  the  more 
experienced  they  are,  the  more  likely  they  are  to  be 
described  as  a  successful  soldier  (9:37). 

Campbell  et  al.  conducted  a  comprehensive  study  that 
reviewed  various  definitions  of  organizational 
effectiveness.  They  described  the  lack  of  a  central 
"definition  and  conceptualization  of  organizational 
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effectiveness"  and  undertook  an  exhaustive  literature  search 

to  uncover  the  principle  dimensions  and  variables  of 

organizational  effectiveness  (2:xii).  Their  final  product 

was  a  catalogue  of  existing  methods  to  measure  effectiveness 

and  a  discussion  of  the  strengths  and  weaknesses  of  each 

method.  The  authors  determined  that  there  were  two  general 

models  under  which  all  approaches  to  measuring 

organizational  effectiveness  fall:  the  goal  centered  model 

and  the  natural  systems  model  (2:5).  Because  the  goal 

centered  model  is  more  applicable  to  the  military 

organization  and  philosophy,  only  it  will  be  defined  here. 

The  goal  centered  model  assumes  that: 

the  organization  is  in  the  hands  of  a  rational  set  of 
decision  makers  who  have  a  set  of  goals  in  mind  they 
wish  to  pursue.  Further,  these  goals  are  few  enough  in 
number  to  be  manageable,  and  can  be  defined  well  enough 
to  be  understood.  (2:6) 

The  authors  then  cite  the  Operational  Research  Model  as 
being  the  most  important  sub-class  of  goal  centered 
techniques  for  measuring  operational  effectiveness.  They 
also  present  the  research  of  Hayward  who  argued  that  the 
most  feasible  approach  to  predict  a  unit's  combat  readiness 
must  include  the  use  of  existing  data  on  the  unit,  theory, 
and,  most  importantly,  expert  judgement  (2:31).  Because  the 
Delphi  technique  is  a  methodology  of  building  a  consensus  of 
opinion  among  experts,  Hayward's  theory  tends  to  support 
using  a  Delphi  approach  in  this  research  on  exercise 
effectiveness  and  combat  readiness. 
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Summary 


This  review  presents  existing  literature  that  explores 
exercise  realism  and  combat  readiness.  Although  realism  is 
recognized  as  being  important  in  training  military  forces 
for  combat,  little  information  is  available  that  describes 
how  the  elements  of  realism  contribute  to  combat  readiness. 
Clearly,  this  is  an  area  that  requires  further  research. 
Chapter  III  describes  the  methodology  used  in  determining 
the  relationship  between  exercise  realism  and  combat 
readiness . 
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UL _ Methodology 


Introduction 

The  Delphi  technique  was  chosen  as  the  methodology  to 
determine  the  relationship  between  exercise  realism  and 
exercise  effectiveness.  The  Delphi  is  a  technique  for 
building  a  consensus  of  opinion  among  experts  in  a 
particular  area  of  interest.  The  procedure  involves  three 
attributes  that  distinguish  the  method  from  other  group 
interaction  methods:  anonymity,  iteration  with  controller 
feedback,  and  statistical  group  response  (24:1).  With  a 
Delphi  sequence,  the  group  members  maintain  their  anonymity, 
and  group  interaction  occurs  through  responses  to 
questionnaires.  Several  iterations  or  rounds  of  the 
questionnaires  are  sent  to  the  participants,  and  summaries 
of  previous  responses  are  included  as  feedback  to  help  build 
a  consensus  among  panel  members.  The  Delphi  procedure 
presents  a  statistical  response  which  includes  the  opinions 
of  the  entire  group.  This  chapter  describes  the  evolution 
of  the  Delphi  technique,  some  advantages  and  disadvantages 
of  Delphi,  and  how  the  method  was  applied  in  this  research. 

The  Evolution  of  Delphi 

The  Delphi  concept  may  be  viewed  as  a  spinoff  of 
defense  research.  "Project  Delphi"  was  the  name  given  to  an 
Air  Force-sponsored  Rand  Corporation  study  that  began  in  the 
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1950s.  The  objective  of  this  study  was  to  estimate ,  from 
the  Soviet  strategic  planner's  point  of  view,  the  effect  of 
atomic  bombs  on  the  United  States  industrial  systems 
complex.  The  objective  of  the  original  study  was  to  "obtain 
the  most  reliable  consensus  of  opinion  cf  a  group  of  experts 
by  a  series  of  intensive  questionnaires  interspersed  with 
controlled  feedback"  (22:10).  The  justifications  for  this 
original  study,  the  inability  tc  collect  accurate  data  and 
the  subjectiveness  of  the  material,  are  still  valid  for  many 
Delphi  applications  today  (22:10). 

The  Delphi  technique  did  not  gain  recognition  outside 
the  defense  community  until  the  1964  Rand  paper,  "Report  on 
a  Long-Range  Forecasting  Study,"  by  T.  J.  Gordon  and  Olaf 
Helmer.  The  study  explored  the  methodological  aspects  of 
the  technique  and  formed  the  foundation  for  a  number  of 
individuals  to  begin  experimentation  with  Delphi  in 
non-defense  areas  (22:10).  Today,  the  Delphi  technique  has 
become  a  fundamental  tool  for  those  in  the  area  of 
technological  forecasting  and  is  used  in  many  technological 
corporations.  The  technique  is  also  used  in  the  classical 
management  science  and  operations  research  where  there  is  a 
need  to  incorporate  more  subjective  information  into 
evaluation  models  (22:11). 


Delphi  Procedure 

Figure  1  illustrates  the  Delphi  method  in  detail.  As 
in  most  methodologies,  the  first  step  is  to  define  the 
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research  problem.  Then,  the  experts  are  selected  and 
questionnaires  are  prepared  that  address  the  investigative 
questions  introduced  for  the  research.  The  questionnaires 
are  distributed  to  the  participants,  and  the  responses  are 
analyzed  to  determine  if  consensus  has  been  reached.  Non¬ 
consensus  items  and  opinion  data  are  provided  as  feedback  to 
the  participants  in  successive  rounds  of  the  survey.  The 
rounds  continue  until  a  consensus  is  reached.  Final  results 
of  the  research  should  be  provided  to  the  panel  members 
( 30:90) . 


Advantages  of  Delphi 

One  of  the  advantages  of  using  the  Delphi  process  is 
reduction  of  negative  effects  associated  with  large  group 
decision  making.  Round  table  discussions  among  experts, 
with  the  objective  of  forming  a  group  position,  often  lead 
to  a  compromise  between  divergent  views.  These  compromises 
may  result  from  "persuasion  by  the  member  with  the  greatesc 
authority,  unwillingness  to  abandon  publicly  expressed 
opinions,  and  the  bandwagon  effect  of  majority  opinion" 
(15:120).  In  his  research  on  the  Delphi  process,  Preble 
found  that  the  method  overcomes  negative  effects  of  group 
interaction  by  providing  structured  format,  systematic 
feedback,  and  anonymity  (29:75).  There  are  also  some 
disadvantages  to  using  the  Delphi  technique. 
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Flowchart  of  the  Typical  Delphi  Process 
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Figure  1.  Delphi  Process  Flowchart 
(30:80) 


Disadvantages  of  Delphi 


One  major  disadvantage  when  using  the  Delphi  technique 
is  time.  Delphi  studies  can  be  extremely  time  consuming 
depending  on  the  number  of  iterations  used  to  develop  group 
consensus  and  the  amount  of  data  that  roust  be  collated  and 
distributed  to  participants  in  each  round.  Identifying  and 
contacting  the  individuals  who  meet  the  criteria  as  an 
expert  in  the  subject  area  can  also  be  a  time  consuming 
process  (19:47). 

Justification  of  the Delphi  Method 

Anonymity.  The  Delphi  technique  eliminates  one  of  the 
disadvantages  of  traditional  group  decision  making  by 
allowing  participants  to  change  their  mind  without  "publicly 
admitting  it  and  losing  face"  (24:20).  Some  variations  of 
Delphi  actually  allow  participants  to  interact;  however,  for 
this  research  anonymity  was  used  to  eliminate  possible 
influences  from  status,  rank,  or  job  position. 

Effectiveness .  H.  W.  Landford  cites  the  effectiveness 
of  the  Delphi  process: 

The  consensus  reflects  reasoned,  self-aware  opinions, 
expressed  in  the  light  of  the  opinions  of  associated 
experts.  Thus,  these  predictions  should  provide 
sounder  basis  for  long-range  decision  making  than  do 
unarticulated  intuitive  judgements.  (21:22) 

Norman  C.  Dalkey,  one  of  the  developers  of  the  Delphi 

technique,  suggests  that  the  Delphi  has  valuable  side 

products : 
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The  Delphi  procedure  is  one  of  the  most  efficient  I 
know  for  uncovering  the  implicit  models  that  lie  behind 
opinions  in  the  "soft"  areas.  One  of  the  most  valuable 
side-products  of  a  Delphi  exercise  concerned  with 
strategic  bombing  was  the  skeleton  of  a  model  which  was 
later  fleshed  out  in  detail.  (6:9) 


Accuracy .  Research  supports  that  a  group  prediction  of 

a  panel  generally  will  be  superior  to  those  obtained  from 

individual  participants.  In  other  words,  "two  heads  are 

better  than  one"  (26:174).  Research  on  Delphi  accuracy  by 

Frederick  Parente  et  al.  indicated  that  "predictions  derived 

from  the  group  were  more  accurate  than  those  of  95  percent 

of  the  individual  panelists,  but  did  not  exceed  in  accuracy 

the  best  panelists"  (26:173). 

Reliability.  A  survey  instrument  is  reliable  to  the 

degree  that  it  supplies  consistent  results  (10:185).  The 

Delphi  technique  is  reliable  if  essentially  the  same  results 

occur  if  the  forecasting  effort  is  replicated,  either  by  the 

same  director  or  by  another  (24:48).  Joseph  Martino's 

research  on  Delphi's  reliability  concludes: 

...with  a  panel  of  larger  than  15,  consisting  of  a 
cross  section  of  experts  in  the  given  field,  it  is 
highly  unlikely  that  another  equally  expert  panel  will 
produce  a  radically  different  median ...  forecasts 
produced  by  the  Delphi  procedure  are  reliable;  that  is, 
different  panels  will  tend  to  produce  about  the  same 
results.  (24:20) 

The  Delphi  technique  was  used  in  this  research  because 
of  the  technique's  accuracy  and  reliability.  The  lack  of 
accurate,  quantifiable  data  concerning  exercise  realism  and 
the  subjective  nature  of  the  research  area  make  the  Delphi 
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technique  an  appropriate  methodology  for  answering  the 
investigative  questions  discussed  in  Chapter  I. 

Delphi  Panel  Selection 

Selecting  members  of  the  expert  panel  is  an  important 
step  in  the  Delphi  sequence.  The  panel  members  must  be 
qualified  in  the  subject  area,  and  the  panel  must  be 
representative  of  the  research  population.  Determining  who 
is  an  expert  can  be  accomplished  in  several  ways,  including 
discrimination  through  educational  credentials  or  past 
accomplishments  (3:140).  For  this  research,  an  expert  is 
defined  as  one  who  has  at  least  four  years  experience  in  an 
Air  Force  job  involving  mobility,  exercise  plans,  or  another 
exercise-related  career  field.  All  experts  selected  for  the 
panel  exceeded  this  criteria,  and  the  panel's  average  work 
experience  exceeded  ten  years. 

The  research  objective  is  to  gain  a  macro  view  of  the 
relationship  between  exercise  realism  and  exercise 
effectiveness.  Thus,  to  be  representative  of  the  Air  Force, 
the  panel  must  represent  several  career  fields,  management 
levels,  and  major  commands.  With  the  help  of  planners  at 
Headquarters  USAF,  twenty-four  individuals  were  contacted 
and  asked  to  participate  in  the  research.  Figure  2  lists 
the  current  or  previous  job  experience  of  each  panel  member 
and  their  current  management  level.  The  experts  were  chosen 
from  the  following  major  commands:  Air  Mobility  Command 
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(AMC),  Air  Combat  Command  (ACC),  and  Air  Force  Special 
Operations  Command  (AFSOC).  Panel  members  were  also 
selected  from  Headquarters  USAF.  Appendix  A  contains  a  list 
of  all  questionnaire  recipients. 


Job  Title/Experience 
Exercise  Evaluation  Team 
Inspector  General 
Operations  Planner 
Security  Police 
Services 

Logistics  Planner 


Management  Level  Humber 

Wing  l 

HQ  USAF/MAJCOM  4 

HQ  USAF/MAJCOM/ Wing  7 

HQ  USAF/MAJCOM/Wing  3 

MAJ COM /Wing  2 

HQ  USAF/MAJCOM/Wing  7 

24 


Figure  2.  Distribution  of  Delphi  Panel  Members 
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Consensus  refers  to  the  general  level  of  agreement 
within  the  expert  panel  and  has  been  defined  in  previous 
Delphi  studies  as  having  at  least  two-thirds  of  the 
respondents  in  agreement  (5:52;  3:146-147).  Therefore, 
consensus  is  defined  in  this  study  as  66.7  percent  or  higher 
agreement.  This  level  is  somewhat  arbitrary,  but  it  is  high 
enough  to  be  considered  a  general  agreement. 

Number  of  Rounds 

There  is  no  clear  answer  to  the  number  of  iterations 
that  should  be  performed  using  the  Delphi  process.  The 
general  finding  is  that  "by  the  end  of  four  rounds,  the 
panel  has  reached  as  much  agreement  as  it  is  ever  going  to 
reach"  (24:27).  Martino  continues  by  stating  that  two 
rounds  may  clarify  the  issues  even  if  full  agreement  is  not 
reached  by  the  panel.  He  also  states  that  a  good  rule  of 
,.1-nb  is  three  rounds.  For  this  study,  consensus  among 
e  .ists  was  obtained  after  two  rounds;  thus,  additional 
iterations  were  not  required. 

Questionnaire  cgnstruction 

The  Delphi  participants  were  mailed  a  series  of 
questionnaires.  Three  types  of  questions  were  used  to 
collect  target  data.  They  were  either  open-ended,  based  on 
a  Likert  scale,  or  based  on  a  ranking  of  options.  The 
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open-ended  questions  were  used  in  the  first  Delphi  round  to 
gain  information  on  levels  of  realism  and  the  relationship 
between  realism  and  combat  readiness.  The  open-ended  format 
was  useful  because  prior  knowledge  of  possible  responses  to 
some  questions  was  unknown.  This  format  also  allowed  for 
capture  of  more  opinion  data  from  the  participants. 

The  second  questionnaire  was  a  reiteration  of  the  first 
questionnaire  and  used  Likert  Scale  questions  to  develop 
panel  consensus.  The  questionnaire  was  constructed  and  sent 
to  those  experts  who  responded  to  the  first  questionnaire. 
Summary  statistics  of  responses  from  the  first  questionnaire 
were  included  as  feedback  for  the  experts.  This  was  done  in 
an  attempt  to  achieve  a  higher  level  of  consensus  by 
exposing  respondents  to  the  opinions  of  other  experts.  The 
full  text  of  both  questionnaires  is  contained  in  Appendix  B. 

Reliability 

The  reliability  of  an  instrument  is  the  degree  that  it 
supplies  consistent  results  (10:187).  For  this  research, 
reliability  is  determined  primarily  by  observing  the 
internal  consistency  of  answers  in  the  survey.  Each 
investigative  question  was  addressed  by  two  distinct 
questions.  If  the  survey  instrument  is  reliable,  an 
individual  expert's  response  to  these  two  questions  should 
be  similar.  The  reliability  can  be  measured  by  the 
correlations  between  the  answers  to  the  two  questions. 
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Validity  is  the  "extent  to  which  a  test  measures  what 
we  actually  wish  to  measure'  (30:180)*  To  improve  validity, 
questionnaires  were  pretested  on  five  AFTT  graduate  students 
wfo  had  previous  experience  as  a  mobility  officer  or 
exercise  planner.  The  students  were  csked  to  evaluate  the 
questions  in  terms  of  clarity,  logic,  and  readability. 

Their  feedback  was  used  in  the  design  of  the  questionnaires. 

LiKert . Seisin 

Questions  using  a  Likert  scale  were  designed  to  measure 
a  range  of  opinion  on  a  specific  question.  Depending  on  the 
questions,  two  variations  of  the  Likec  Scale  were  used. 
Figure  3  shows  these  two  options  (5:47). 


Likert  Scales  used  in  the 

Questionnaire 

i 

a . 

b. 

c . 

d . 

e . 

Strong! y 

Neither  Agree 

Strongly 

Cisagree 

Disagree 

Nor  Cisagree 

Agree 

Agree 

a . 

b. 

c. 

d. 

e . 

Very 

Neither  High 

Very 

Low 

Low 

Nor  Low 

High 

High 

iqure  3.  Likert  Scales  Used  in  the  Questionnaires 
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The  advantage  of  using  the  Likert  Scale  is  that  the 
difference  in  qualitative  responses  can  be  quantified 
(6:220).  Mean  responses  to  each  question  were  computed  and 
group  consensus  was  determined  by  the  percent  agreement  on  a 
Likert  scale  response.  Individual  responses  of  "highly 
agree"  and  "agree"  were  grouped  as  were  "highly  disagree" 
and  "disagree".  This  provided  a  measure  of  general 
agreement  or  disagreement  among  the  experts  and  increased 
the  probability  of  panel  consensus. 

Some  questions  required  respondents  to  rank  order 
factors  that  affect  exercise  realism.  To  obtain  the  overall 
ranking  for  a  particular  response,  the  frequency  of  each 
response  and  the  average  ranking  for  the  response  among  all 
panel  members  was  calculated.  The  product  of  these  numbers 
was  multiplied  by  a  weighting  factor  to  obtain  the  overall 
ranking  for  the  response.  Figure  4  depicts  the  weighted 
ranking  system.  This  scale  was  used  because  it  ensured  that 
the  factors  ranked  higher  by  panel  members  were  given  the 
qreatest  weight.  Additionally,  this  system  accounts  for 
those  responses  that  received  low  rankings  by  panel  members 
yet  were  listed  by  several  respondents.  None  i»e  experts 

responded  with  more  than  eight  factors. 
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Weighted  Ranking  System 


Average  Ranking 

1st 

2nd 

3rd 

4  th 

5  th 
6th 
7th 
3th 


Weighted  Seale 

1.00 

0.875 

0.750 

0.625 

0.50 

0.375 

0.250 

0.175 


Figure  4.  Weighted  Ranking  System 


The  Delphi  technique  is  a  sat  of  procedures  used  to 
gather  a  consensus  of  expert  opinion.  The  first  part  of 
this  chapter  described  the  evolution  of  the  Delphi  process, 
some  of  the  advantages  and  disadvantages  of  the  technique, 
and  the  justification  for  using  Delphi  in  this  research. 

The  remaining  part  of  the  chapter  outlined  the  Delphi 
panel  selection  process  and  the  issues  involved  in 
constructing  and  administering  the  questionnaires.  Chapter 
IV  explains  how  the  data  from  the  Delphi  surveys  were 
analyzed  and  used  to  answer  the  investigative  questions 
introduced  in  Chapter  I. 
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The  purpose  of  this  research  was  to  determine  the 
relationship  between  exercise  realism  and  exercise 
effectiveness.  The  Delphi  method  was  used  to  determine  the 
current  levels  of  exercise  realism,  the  factors  that  affect 
realism,  and  how  these  factors  impact  unit  combat  readiness. 
This  chapter  analyzes  the  questionnaire  data  submitted  by 
the  Delphi  panel  experts.  Survey  results  from  rounds  one 
and  two  of  the  Delphi  sequence  are  analyzed  and  discussed. 

As  defined  in  Chapter  III,  consensus  is  achieved  with  a 
66.7  percent  or  higher  agreement  on  a  response.  For  Likert 
scale  questions,  any  agreement  of  66.7  percent  or  higher  on 
one  side  of  the  agree/disagree  axis  is  considered  a 
consensus.  The  complete  set  of  survey  responses  is  included 
in  Appendix  C. 

The  TicsS  Sm-vey 

The  first  round  questionnaire  (Appendix  B)  was  sent  to 
24  experts.  The  response  rate  was  83.3%  with  20  of  24 
panelists  participating  in  the  survey.  The  survey  contained 
several  open-ended  and  Likert  scale  questions  that  addressed 
the  investigative  questions  introduced  in  Chapter  I. 

Several  questions,  such  as  those  concerning  factors  that 
impact  exercise  realism,  were  exploratory  in  nature.  These 
questions  were  required  so  an  accurate  list  of  factors  could 
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be  established  and  presented  to  the  panelists  in  the  second 
round  of  the  Delphi  process.  Consensus  among  the  experts 
was  not  achieved  for  several  first  round  questions. 

Analysis  of  responses  from  these  questions  was  useful  in 
developing  the  second  survey. 

The  first  survey  addressed  six  general  areas: 
background  data  on  the  experts,  current  levels  of  exercise 
realism,  exercise  realism  and  combat  readiness,  factors  that 
affect  realism,  factors  that  affect  combat  readiness,  and 
advantages  and  disadvantages  cf  increasing  exercise  realism. 

Background  Data.  The  first  four  questions  on  the 
survey  were  background  questions  and  addressed  the  expertise 
of  the  panel  members.  Question  #2  requested  the  panelists' 
number  of  years  of  work  experience  involving  military 
exercises.  The  results  are  shown  in  Table  1.  The  mean  of 
3.45  indicates  that  the  average  work  experience  of  the 
experts  was  between  10  and  12  years.  None  of  the  experts 
had  less  than  six  years  of  experience.  Thus,  according  to 
the  criteria  established  in  Chapter  III,  all  twenty 
panelists  qualified  a.  experts  for  this  Delphi  study.  The 
data  on  operational  experience  was  additional  verification 
of  the  qualifications  of  the  Delphi  participants.  Seventeen 
of  the  twenty  panelists  (85%)  had  participated  in  at  least 
one  operational  mission.  For  the  group  of  experts  with 
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operational  experience,  the  average  number  of  missions  they 
had  participated  in  was  5.8. 

Table  1 

Delphi  Panel  Background  Data 
(Delphi  Round  One) 


1.  Years  of  Work  Experience 

Response  ( years ) 

2-6 

6-10 

10-12 

>12 


Er..e.q.u£ngy 

o 

3 

5 

12 


Mean 

3.45 


2.  Participation  in  Operational  Missions 


Response 


Percent. participating  Humber  of  missions 


Yes  17  85%  (17/20) 

NO  3 


Min  1 

Max  25 

Mean  5.8 


Levels  of  Realism.  Several  questions  were  designed  to 
determine  the  current  level  of  realism  in  Air  Force 
exercises.  Question  #5  asked  panelists  to  rate  the  general 
level  of  exercise  realism.  The  Likert  scale  responses  are 
shown  in  Table  2. 
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Table  2 


Responses  to  Question  #5 
(Delphi  Round  One) 


Topic 


Frequencies 

1  __  2 _ 3 _ 4 _ £  Mfian 


Consensus 


Current  level  06  8  6  0  3.0  Ho 

of  realism 


Legend  for  Frequencies 

1  =  Strongly  Disagree 

2  =  Disagree 

3  =  Neither  Agree  nor  Disagree 

4  =  Agree 

5  =  Strongly  Agree 


Because  there  was  not  66.7%  or  higher  agreement  on  a 
response,  there  was  not  a  consensus  among  the  experts 
concerning  current  levels  of  realism.  Most  panelists 
believed  that  exercises  were  neither  realistic  nor 
unrealistic  (40%). 

One  of  the  objectives  of  this  research  was  to  determine 
if  a  difference  in  realism  existed  among  major  commands  and 
management  levels.  Questions  #6  and  #7  were  designed  to 
explore  these  issues.  The  experts  were  asked  to  list  the 
major  commands  to  which  they  had  been  assigned  and  to 
identify  which  command,  if  any,  conducted  the  most  realistic 
exercises . 
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The  summary  of  responses  is  shown  in  Table  3.  The 
experts  were  grouped  according  to  their  major  command  and 
management  level  to  highlight  particular  variances  in 
responses.  The  "Number  in  Group"  column  shows  the  number  of 
experts  in  a  particular  "Group.”  The  "Responses"  indicate 
the  number  of  experts  who  voted  for  a  major  command  as 
conducting  the  most  realistic  exercises.  Some  experts 
listed  more  than  one  major  command  if  they  believed  the 
commands  conducted  more  realistic  exercises  than  other 
commands  but  were  equal  to  each  other.  Table  4  indicates 
the  number  of  responses  for  each  command  when  grouping  the 
experts  by  management  level. 


Table  3 

Responses  to  Question  #7  (Delphi  Round  One) 


Number  in 


_ Qrg>  yp _  fitSMP 

MAJCOM 

ACC  wing  3* 

ACC  MAJCOM  4* 

AFSOC  Wing  l 

AFSOC  MAJCOM  3 


Responses 

ACC  AFSOC  AMC  PACAF  USAFE 
2 

2  1 

1 

3 


AMC  Wing 
AMC  MAJCOM 


3 

3* 


1 


1 


1 

1 


1 


HQ  USAF 


3  1 


3  1 


*  indicates  one  expert  in  the  group  did  not  respond  or  did 
not  indicate  a  difference  in  realism  among  major  commands. 


Table  4 


Total  Responses  by  Management  Level 


Management 

Level 

Total 

Experts 

Responses 

ACC 

AFSOC  AMC  PACAF 

USAFE 

Wing 

7 

2 

2  11 

MAJCOM 

10 

3 

3  2 

HQ  USAF 

3 

1 

3 

1 

TOTAL 

20 

6 

5  3  4 

1 

Of  thoce  experts  who  belonged  to  a  major  command,  10  of 
17  voted  for  tha  command  tc  whicn  they  are  currently 
assigned.  Th^s  phenomena  war  most  pronounced  among  AFSOC 
members  where  }00%  (4  of  4)  believed  AFSOC  conducted  the 
most  realistic  exercises.  57%  (4  of  7)  of  ACC  members  voted 
for  their  major  command  and  only  33%  (2  of  6)  of  members 
assigned  to  AMC  believed  AMC  conducted  the  most  realistic 
exercises.  All  panelists  from  HQ  USAF  believed  PACAF 
conducted  the  most  realistic  exercises  (3  of  3)  although  one 
believed  the  exercises  conducted  by  USAFE  and  CENTAF  (a 
Numbered  Air  Force  under  ACC)  were  equally  realistic. 

There  was  no  apparent  difference  of  opinion  among 
MAJCOM  and  wing  staff  experts.  In  fact,  the  groupings 
closely  paralleled  each  other.  Two  of  three  ACC  wing 
members  believed  ACC  conducted  the  most  realistic  exercises. 
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and  two  of  four  experts  from  HQ  ACC  expressed  the  sane 
belief.  Of  the  AMC  wing  experts,  one  of  three  believed  AMC 
was  the  most  realistic.  This  sane  distribution  occurred 


among  AMC  major  command  experts. 

Applying  an  arbitrary  scaling  technique  based  on  the 
experts'  depth  (years  of  exercise  experience)  and  breadth 
(number  of  commands  served  in)  of  experience  is  one  method 
to  analyze  the  differences  between  the  ratings  of  the  major 
commands.  This  technique  was  applied  to  each  expert's  vote 
to  obtain  an  average  rating  for  each  major  command.  The 
explanation  of  the  scaling  technique  and  its  application  is 
shown  in  Appendix  D.  A  summary  of  the  average  ratings  for 
each  major  command  is  shown  in  Table  5. 


Table  5 

Ratings  of  Exercise  Realism  Among  Major  Commands 


MAsLCQM  AVERAGE  -BAlIMfi 


ACC 

PACAF 

AFSOC 

USAFE 

AMC 


.  848 
.834 
.763 
.750 
.723 


The  limitation  of  this  type  of  ranking  is  obvious. 
Because  the  distribution  of  factors  is  arbitrary,  the 
resultant  ranks  are  subjective.  Using  a  different 
distribution  of  factors  could  yield  very  different  results. 
Nevertheless,  it  is  interesting  to  note  that  the  major 
command  which  received  the  least  amount  of  support  from  its 
own  members  ranks  near  the  bottom  in  this  design. 

The  relatively  low  ranking  of  AFSOC  can  be  explained 
because  AFSOC  expert  representation  is  lowest  among  the 
major  commands.  Further,  AFSOC  members  were  characterized 
by  a  relatively  low  breadth  of  experience  factor.  This, 
along  with  the  fact  that  very  few  of  the  other  experts  had 
any  AFSOC  experience,  tended  to  lower  the  value  of  the  AFSOC 
ranking. 

Exercise.  Realism  .and  .Combat  Readiness.  The  majority  of 
questions  on  the  first  survey  were  designed  to  determine  the 
factors  that  affect  exercise  realism  and  to  analyze  how 
these  factors  contribute  to  unit  combat  readiness.  Question 
#8  addressed  whether  an  exercise  must  be  realistic  in  order 
to  be  effective.  In  other  words,  must  an  exercise  be 
realistic  to  improve  the  readiness  of  combat  units?  The 
responses  are  shown  in  Table  6. 
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Table  6 


Responses  to  Question  #8 
(Delphi  Round  One) 


IffiRjg 

Exercise 

Effectiveness 


Frequencies 

_2 _ 3  4 _ 5  Mean 

3  0  6  11  4.25 


Yes  (85%) 


Legend  for  Frequencies 

1  =  Strongly  Disagree 

2  =  Disagree 

3  =  Neither  Agree  nor  Disagree 

4  =  Agree 

5  =  Strongly  Agree 


Consensus  was  reached  on  this  question.  85.0%  of  the 
respondents  agreed  or  strongly  agreed  that  an  exercise  roust 
be  realistic  to  be  effective. 

Factors  Affecting  Realism.  Question  #9  was  an  open- 
ended  question  that  asked  the  experts  to  list  and  rank  at 
least  three  factors  that  affect  the  amount  of  realism  in  an 
exercise.  There  were  a  variety  of  responses  to  this 
question  and,  as  expected,  no  consensus  was  reached  among 
the  panel  members.  Twenty  different  factors  were  listed  by 
the  experts.  The  weighted  ranking  scale  was  applied  as 
described  in  Chapter  III  to  determine  an  overall  ranking  for 
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each  factor.  Table  7  summarizes  the  results  of  the 
analysis.  The  factors  receiving  the  highest  overall  score 
are  listed  first.  Appendix  D  (Table  16)  illustrates  how  the 
score  for  each  factor  was  computed. 


Table  7 

Factors  Affecting  Realism 
(Delphi  Round  One) 


1.  Availability  of  personnel  (proper  force  mix) 

2.  Exercise  location  (weather,  terrain,  off-base  location) 

3.  Scenario  development  (realistic  threat) 

4.  Funding 

5.  Availability  of  airlift  resources 

6 .  Time 

7.  Availability  of  Base  Operating  Support  (BOS)  equipment 

8.  Number  of  simulations/assumptions 

9.  Enthusiasm/attitude  of  participants 

10.  Airspace  limitations/availability  of  training  areas 

11.  Senior  leadership  involvement/emphasis 

12.  Other  Wing  priorities  (day-to-day  activities, 
operational  missions) 

13.  Exercise  size 

14.  Environmental  factors  (noise,  local  community 
restrictions) 

15.  Safety  concerns/restrictions 

16.  Difficulties  in  deploying  as  a  whole  unit 

17.  Excessive  management  control  preventing  free-play 
exercise 

18.  Pre-exercise  planning 

19.  Reluctance  to  allow  "operations"  to  fail  due  to  support 
function  shortfalls 

20.  Lack  of  training  in  re-deployment  procedures 
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Using  this  ranking  system,  the  five  roost  important 
factors  that  impact  realism  are:  availability  of  personnel, 
exercise  location,  scenario  development,  funding,  and 
availability  of  airlift  resources.  The  complete  list  of 
factors  was  used  in  the  second  round  survey  so  panelists 
could  rate  the  importance  of  factors  listed  by  other 
experts . 

Factors  Affecting  Combat  Readiness.  Question  #10  was 
similar  to  the  previous  question,  but  it  was  designed  to 
determine  how  the  factors  contributed  to  an  exercise's 
effectiveness.  The  experts  were  asked  to  use  their  list  of 
factors  submitted  in  question  #8  and  to  rank  the  factors 
that  are  roost  important  to  an  exercises's  effectiveness.  In 
other  words,  which  factors  are  necessary  to  ensure  that  an 
exercise  improves  combat  readiness.  As  in  question  #9, 
several  different  factors  were  submitted.  The  weighted 
ranking  system  was  applied  to  obtain  an  overall  score  for 
each  factor.  Appendix  D  (Table  17)  contains  the  list  of 
factors  and  computation  of  scores.  Table  8  summarizes  the 
analysis . 
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Table  8 


Factors  Most  important  to  Combat  Readiness 
(Delphi  Round  One) 


1.  Exercise  location  (weather,  terrain,  off-base) 

2.  Availability  cf  Personnel(  proper  force  mix,  host 
nation  participation) 

3.  Number  of  simulations/assumptions 

4 .  Funding 

5.  Scenario  development  (proper  exercise 
obj  ;tive/realistic  tnreat) 

6.  Airspace  liroitations/availability  of  training  areas 

7.  Availability  of  Base  Operating  Support  (BOS)  equipment 

8.  Availability  of  airlift  resources 

9.  Senior  leadership  involveraent/emphasis 

10.  Enthusiasm/attitu'Je  of  participants 

11.  Exercise  size 

.  Reluctance  to  allow  "operations"  to  fail  due  to 
‘•support"  function  shortfalls 


The  five  factors  most  important  in  improving  combat 
readiness  were  location,  availability  of  personnel,  number 
of  simulations/assumptions,  funding,  and  scenario 
development.  Four  of  the  five  factors  were  included  in  the 
top  five  factors  that  affect  exercise  realism.  These 
factors  were  used  in  the  second  survey  in  an  attempt  to 
develop  a  consensus  opinion  among  the  panelists. 

Advantages  and  Disadvantages  of  Realistic  Exercises. 
Questions  #11  and  #12  asked  the  experts  to  provide  some 
advantages  and  disadvantages  of  increasing  realism  in 
exercises.  The  list  of  all  advantages  and  disadvantages 
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submitted  by  the  experts  is  shown  in  Figure  5.  Host 
panelists  listed  only  one  advantage  and  disadvantage,  and 
none  of  the  experts  listed  all  of  them.  Thus,  to  determine 
if  a  consensus  can  be  reached  on  the  advantages  and 
disadvantages  of  increasing  exercise  realism,  the  experts 
were  asked  to  respond  to  the  complete  lists  on  the  second 
survey . 


Advantages : 

1.  Improved  unit  readiness  (cohesion,  confidence, 
adaptability) 

2.  Useful  in  identifying  weaknesses  in  current 
training  and  doctrine 

3.  Effective  assessment  of  equipment  capabilities 

4.  Improved  decision  making  and  innovative  thinking 

Disadvantages : 

1.  Increased  cost  (longer  exercises  and  increased  use 
of  resources) 

2.  Increased  safety  mishaps  (equipment  and  personnel 
loss) 

3.  Increased  hardships  on  families 

4.  Disruption  of  peacetime  missions. 


Figure  5.  Advantages  and  Disadvantages  of  Increasing 
Exercise  Realism 


The  final  question  on  the  first  questionnaire  was 
designed  to  explore  the  benefits  of  increasing  realism  in 
exercises.  The  experts  were  asked  if  a  point  exists  where 
increasing  realism  fails  to  provide  any  additional  benefits. 
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Consensus  was  reached  on  this  question  with  70%  of  the 
panelists  agreeing  that  there  is  a  point  beyond  which 
little  can  be  gained  by  continuing  to  increase  realism. 

Ci  iments  included  responses  such  as  "once  an  understanding 
of  key  elements  in  the  task  is  achieved,  diminishing  returns 
apply."  However,  30%  of  the  panelists  believed  that 
increases  in  realism  are  always  beneficial.  A 
representative  response  of  this  group  is  "an  exercise  that 
is  not  as  realistic  as  it  could  be  seems  illogical"  and 
"increasing  realism  in  an  exercise  will  make  the  exercise 
more  like  a  combat  situation  and  improve  combat  readiness." 

Table  9 

Responses  to  Question  #8 
(Delphi  Round  One) 


Topic 


Frequencies 

1 _ Z  Mean 


Benefits  of  increased  realism  14  6  1.3 


consgnsns 

Yes  (70%) 


Legend  for  Frequencies 


1 

2 


Yes 

No 


The  Second  Survey 


The  second  round  questionnaire  (Appendix  B)  was  sent  to 
the  20  experts  who  responded  to  round  one.  The  response 
rate  was  70%  with  14  of  20  panelists  participating  in  the 
survey.  The  lower  response  rate  resulted  from  early 
retirements  and  extended  deployments  of  some  of  the  experts. 
The  survey  addressed  areas  from  the  first  questionnaire 
where  the  panelists  did  not  reach  consensus.  The  following 
areas  were  addressed:  factors  affecting  realism,  factors 
affecting  combat  readiness,  current  levels  of  realism,  and 
advantages  and  disadvantages  of  increasing  exercise  realism. 

Factors  Affecting  Realism.  The  first  survey  asked 
panelists  to  list  and  rank  factors  that  significantly  affect 
exercise  realism.  This  list  of  20  different  factors 
submitted  by  the  experts  was  used  in  question  #4  of  the 
second  survey.  Using  a  Likert  scale,  the  experts  were  asked 
to  indicate  the  extent  to  which  they  agreed  that  each  factor 
was  important  to  exercise  realism.  Thus,  all  the  experts 
responded  to  the  overall  list  of  factors  generated  from  the 
first  survey.  Table  18  of  Appendix  D  shows  the  mean  Likert 
responses  for  the  factors. 

To  develop  a  list  of  factors  considered  most  important 
in  contributing  to  exercise  realism,  the  results  from  both 
Delphi  rounds  were  combined.  An  arbitrary  selection 
procedure  was  used.  The  following  criteria  were  used  to 
select  the  most  significant  factors  affecting  realism: 
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an  overall  score  of  at  least  10.0  from  analysis  of  first 
round  data  (Table  16)  and  a  mean  Likert  response  of  at  least 
4.0  from  second  round  analysis  (Table  18).  The  results  are 
shown  below  in  Table  10. 

Although  this  selection  technique  is  arbitrary,  it 
considers  the  rankings  submitted  by  the  experts  from  the 
first  survey  and  it  also  considers  the  consensus  among  panel 
members  concerning  a  factor's  importance.  A  larger  number 
of  factors  could  have  been  selected,  but  the  intent  was  to 
focus  on  the  factors  that  appear  to  be  the  roost  significant. 
Thus,  the  selection  criteria  attempted  to  group  the  factors 
that  received  high  scores  from  both  surveys. 

Table  10 

Factors  Affecting  Realism 
(Delphi  Round  Two) 


1.  Availability  of  personnel 

2.  Scenario  development 

3.  Funding 

4.  Time 

5.  Availability  of  airlift 

6.  Availability  of  Base  Operating 
Support  (BOS)  equipment 


Factors  Affecting  Combat  Readiness.  Question  #5  was 
similar  to  question  #4,  but  it  addressed  the  12  factors 
affecting  combat  readiness  generated  from  the  first  survey. 
The  experts  used  a  Likert  scale  to  indicate  the  extent  to 
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which  they  agreed  that  each  factor  impacts  unit  combat 
readiness.  The  mean  Likert  responses  for  each  factor  are 
shown  in  Appendix  D  (Table  19). 

The  procedure  used  to  identify  significant  factors 
affecting  realism  was  also  used  for  *’ie  f?  s  that  impact 
combat  readiness.  In  this  case,  the  critt.r:*  used  to  select 
factors  were  an  overall  score  of  at  least  2.0  (from  survey 
one,  Table  17)  and  a  mean  Likert  response  of  at  least  4.0 
(from  survey  two.  Table  19).  Table  11  shows  the  list  of 
factors  that  meet  this  criteria.  More  factors  were  included 
in  this  list  than  in  Table  10  because  there  was  not  an 
obvious  separation  among  overall  scores  of  factors  from  the 
round  one  analysis.  Thus,  a  low  overall  score  (2.0)  was 
used  as  the  cutoff  to  ensure  that  important  factors  were  not 
omitted. 


Table  11 

Factors  Affecting  Combat  Readiness 
(Delphi  Round  Two) 


1.  Exercise  location 

2.  Availability  of  personnel 

3.  Number  of  siraulations/assumptions 

4 .  Funding 

5.  Scenario  development 

6.  Airspace  liroitations/availability  of 
training  areas 

7.  Availability  of  airlift 

8.  Senior  leadership  involvement/emphasis 
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Current  Levels  of  Realism.  A  consensus  was  not  reached 


on  round  one  concerning  the  current  level  of  exercise 
realism.  The  mean  Likert  response  was  about  3.0  indicating 
that  the  experts  believed  that  exercises  were  neither 
realistic  nor  unrealistic.  Thus,  this  subject  was  addressed 
in  round  two.  Question  #2  of  the  second  survey  asked  the 
experts  if  the  current  level  of  realism  in  exercises  is 
adequate.  As  shown  in  Table  12,  over  85%  of  the  panelists 
disagreed  that  Air  Force  exercises  contained  adequate  levels 
of  realism.  Thus,  although  experts  did  not  reach  consensus 
on  the  current  level  of  exercise  realism,  most  agreed  that 
the  current  level  is  not  adequate. 

Table  12 

Responses  to  Question  #2 
(Delphi  Round  Two) 


Topic. 


Frequencies 

l _ 2 _ 2 _ a _ &  Mfeap 


Consensus 


Current  Level  2  10  0  2  0  2 . 14  Yes  (85.7%) 

of  Realism 


Legend  for  Frequencies 

1  =  Strongly  Disagree 

2  =  Disagree 

3  =  Neither  Agree  nor  Disagree 

4  =  Agree 

5  =  Strongly  Agree 
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Advantages  and  Disadvantages  of  Increasing  Realism. 
Question  #3  asked  the  experts  to  respond  to  the  list  of 
advantages  and  disadvantages  developed  from  the  first  survey 
responses.  A  Likert  scale  was  used  to  determine  the  extent 
that  the  experts  agreed  with  each  list.  Tables  13  and  14 
summarize  the  responses.  The  experts  reached  consensus 
concerning  both  lists,  but  they  were  in  less  agreement  with 
the  disadvantages.  Some  experts  wrote  comments  suggesting 
that  some  of  the  disadvantages  apply  to  field  exercises  but 
could  be  reduced  with  command  post  exercises. 


Table  13 

Responses  tc  Question  #3 
(Delphi  Round  Two) 


Frequencies 

lapis.  1 _ 2 _ 2 _ 4 _ 5 

Advantages  of  00059 

Increasing  Realism 


Msait  Canseasns 

4.64  Yes  (100%) 


Legend  for  Frequencies 

1  =  Strongly  Disagree 

2  =  Disagree 

3  =  Neither  Agree  nor  Disagree 

4  =  Agree 

5  =  Strongly  Agree 
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Responses  to  Question  #3 
(Delphi  Round  Two) 


ToDic  1 

Frequencies 

2  3  4  5 

Mean 

Consensus 

Disadvantages  of  0 

Increasing  Realism 

4  0  8  2 

3.57 

Yes  (71.4%) 

Legend  for  Frequencies 

1  =  Strongly  Disagree 

2  =  Disagree 

3  -  Neither  Agree  nor  Disagree 

4  =  Agree 

5  =  Strongly  Agree 


Summary 

This  chapter  described  how  the  data  received  from  the 
Delphi  panelists  were  analyzed.  Two  rounds  of  surveys  were 
used  to  gather  data  that  addressed  the  investigative 
questions  introduced  in  Chapter  I.  The  first  survey 
explored  the  factors  that  impact  exercise  realism,  factors 
that  impact  combat  readiness,  and  the  differences  in 
exercise  realism  among  major  commands  and  management  levels. 
Data  was  also  collected  to  examine  some  advantages  and 
disadvantages  of  increasing  exercise  realism. 

The  second  survey  was  a  reiteration  of  the  first  round 
but  attempted  to  build  consensus  among  panel  members.  The 
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survey  addressed  current  levels  of  realism,  factors 
affecting  realism,  factors  impacting  combat  readiness,  and 
the  advantages  and  disadvantages  of  increasing  realism. 
Consensus  was  reached  on  all  questions.  Chapter  V  discusses 
the  results  of  the  analysis  in  terms  of  the  investigative 
questions  and  presents  research  conclusions  and 
recommendations . 
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General 

This  research  was  designed  to  explore  the  relationship 
between  realism  in  Air  Force  exercises  and  exercise 
effectiveness.  Experts  from  several  career  fields  and 
management  levels  provided  data  that  helped  explain  how 
exercise  realism  contributes  to  unit  combat  readiness.  The 
following  specific  issues  were  analyzed:  current  levels  of 
exercise  realism,  factors  that  impact  realism  and  combat 
readiness,  and  the  advantages  and  disadvantages  of 
increasing  exercise  realism. 

The  existing  literature  on  realism  contains  several 
sources  that  emphasize  the  importance  of  realism  in 
training  forces  for  combat.  However,  the  literature  also 
indicates  that  little  research  has  been  accomplished  that 
specifically  addresses  the  relationship  between  exercise 
realism  and  exercise  effectiveness. 

The  Delphi  Method  was  selected  as  the  methodology  to 
solicit  expert  opinion  and  to  generate  data  on  exercise 
realism.  The  success  of  a  qualitative  approach  of  this  type 
depends  on  the  participation  of  the  subiect  matter  experts. 
The  experts'  enthusiasm  and  persistence  were  indicative  of 
their  interest  in  discussing  issues  involving  Air  Force 
training  exercises. 
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This  chapter  presents  a  discussion  of  the  findings 
resulting  from  this  research  and  explores  the  investigative 
questions  that  were  introduced  in  Chapter  I.  Although  the 
authors  have  tried  to  accurately  reflect  only  those  views  of 
the  expert  panel,  some  conclusions  may  reflect  the  authors' 
bias  from  their  experiences  through  participation  in 
numerous  operational  missions,  readiness  exercises,  and 
deployments.  Each  of  the  investigative  questions  are 
addressed  under  the  appropriate  heading. 

Perceptions  on  Realism 

As  described  in  Chapter  II,  several  military  leaders 
have  emphasized  the  need  for  realism  in  training  combat 
forces;  however,  the  existing  literature  on  exercise  realism 
fails  to  demonstrate  how  realism  affects  combat  readiness 
and  exercise  effectiveness.  In  their  research  on  the 
performance  of  infantry  soldiers,  Edgarton  and  Graham  only 
indirectly  linked  training  realism  to  combat  effectiveness. 
For  this  research  on  Air  Force  exercises,  the  Delphi  experts 
suggest  a  strong  relationship  between  exercise  realism  and 
training  effectiveness.  The  experts  reached  a  high  level  of 
consensus  on  this  issue  with  85  percent  of  the  panelists 
agreeing  that  exercises  must  be  realistic  to  be  effective. 
However,  some  panelists  disagreed  with  the  concept  that 
effective  exercises  require  complete  realism  to  be 
effective.  The  benefits  achieved  through  exercises  depend 
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on  the  specific  training  objectives  and  the  experience  level 
of  participants.  For  example,  some  combat  skills  can  be 
improved  in  a  very  controlled  exercise  that  eliminates  the 
chaos  and  confusion  that  occurs  in  a  highly  realistic 
training  environment. 

While  the  experts  agreed  that  realism  is  important  in 
developing  exercise  effectiveness,  they  did  not  reach 
consensus  on  the  current  level  of  realism  in  Air  Force 
exercises.  Some  experts  believed  that  exercise  realism  was 
too  low,  and  an  equal  number  believed  that  exercise  realism 
was  too  high.  The  roost  common  response  among  the  experts 
was  that  realism  is  neither  too  high  nor  too  low.  The 
experts'  ambivalence  toward  this  subject  suggests  that 
current  exercises  are  not  as  realistic  or  effective  as  they 
could  be. 

Although  the  experts  could  not  agree  on  the  current 
level  of  realism,  they  strongly  agreed  (85%  consensus)  that 
Air  Force  exercises  did  not  contain  adequate  levels  of 
realism.  Thus,  whatever  the  current  level  of  exercise 
realism,  it  is  not  enough  to  optimize  training. 

Surprisingly,  the  experts'  responses  regarding  current 
levels  of  realism  did  not  vary  according  to  their  career 
field,  management  level,  or  major  command.  Inadequate 
funding  for  exercises  is  a  common  theme  in  both  the 
literature  and  the  experts'  responses  and  may  account  for 
the  general  agreement  among  panelists  that  exercise  realism 
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is  inadequate.  Other  factors,  though,  may  certainly 
contribute  to  this  problem. 

While  the  level  of  exercise  realism  is  clearly 
important  in  creating  effective  training,  the  experts 
agreed  that  the  law  of  diminishing  returns  applies.  In 
other  words,  a  point  exists  at  which  increasing  realism 
fails  to  contribute  to  training  effectiveness.  In  fact, 
increased  realism  may  actually  detract  from  the  overall 
effectiveness  of  an  exercise.  For  example,  removing 
simulated  casualties  from  participation  in  an  exercise  may 
simulate  realistic  conditions,  but  this  practice  eliminates 
any  additional  training  the  "casualties"  may  have  gained. 
Other  disadvantages  associated  with  increasing  exercise 
realism,  such  as  the  increased  risk  of  mishaps,  are 
addressed  later  in  the  chapter. 

Levels  of  -Realism  .Among  MAJC9M? 

One  of  this  research's  investigative  questions 
addressed  the  differences  in  realism  among  Air  Force  major 
commands.  The  data  analysis  of  the  first  Delphi  survey 
resulted  in  the  following  ranking  of  MAJCOMs  (MAJCOM  with 
the  most  realistic  exercises  listed  first):  Air  Combat 
Command  (ACC),  Pacific  Air  Forces  (PACAF),  Air  Force  Special 
Operations  Command  (AFSOC),  United  States  Air  Forces  in 
Europe  (USAFE) ,  and  Air  Mobility  Command  (AMC) .  The  experts 
from  the  Air  Staff  universally  selected  PACAF  as  the  command 
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that  conducted  the  most  realistic  exercises.  Many  experts 
suggested  that  the  scope  and  effectiveness  of  the  USAFE  and 
AMC  exercise  programs  have  been  reduced  because  of  funding 
requirements  for  real  world  missions.  AFSOC's  ranking  was 
artificially  lowered  due  to  the  arbitrary  weighting  factors 
used  in  ranking  the  MAJCOMs  (Appendix  D). 

The  methodology  used  to  rank  the  MAJCOMs  appears 
valid.  However,  it  is  difficult  to  compare  MAJCOMs  on  the 
macro-exercise  program  level,  and  the  resulting  information 
is  not  particularly  useful.  As  one  expert  responded, 
"Comparisons  are  invalid;  it/s  comparing  apples  and 
oranges.  Each  command  focuses  on  aspects  important  to  its 
particular  mission."  Perhaps  future  research  can  use  this 
methodology  to  examine  the  satisfaction  level  among  the 
various  functional  a**eas  of  the  different  MAJCOMs.  The 
information  obtained  from  this  type  of  approach  would  be 
specific  enough  to  support  detailed  recommendations. 

The  components  of  Realism 

A  primary  objective  of  this  research  was  to  identify 
factors  that  significantly  affect  exercise  realism.  The 
analysis  of  the  data  supplied  by  the  Delphi  experts 
resulted  in  the  following  six  factors  (highest  ranked  factor 
listed  first):  the  availability  of  personnel  for  exercise 
participation,  scenario  development,  funding  for  exercises, 
tine,  the  availability  of  airlift,  and  the  availability  of 
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base  operating  support  (BOS)  equipment.  Each  of  these 
factors  represents  a  resource  constraint  that  can  limit 
exercise  realism.  Surprisingly,  location,  while  mentioned 
repeatedly  by  the  experts,  did  not  score  high  enough  to 
appear  in  the  final  list.  This  suggests  that  the  experts 
place  a  greater  importance  on  procedural  and  task-oriented 
concerns.  In  other  words,  the  experts  believe  that  how 
tasks  are  done  is  more  important  than  where  they  are  done. 

These  six  factors,  or  constraints,  are  not  mutually 
exclusive  and  interact  in  affecting  exercise  realism.  These 
relationships  are  illustrated  in  the  survey  responses. 

For  example,  according  to  various  experts'  opinions,  the 
removal  of  constraints  on  personnel  availability  allows 
exercise  planners  to  develop  more  complete  and  realistic 
exercise  scenarios.  Exercises  that  follow  true  Joint 
Operational  Planning  and  Execution  System  (JOPES)  planning 
procedures  permit  a  more  realistic  information  and  tasking 
flow  and  more  closely  create  the  "fog  of  war." 

Additionally,  realistic  scenarios  allow  managers  to  identify 
shortfalls  in  communications  systems.  Simulated  airlift 
activities  prevent  training  in  aircraft  loading  and  result 
in  other  simulations  that  further  reduce  exercise  quality. 
Aircraft  availability  also  indirectly  addresses  the  ability 
to  move  and  operate  in  a  location  other  than  the  home  base. 
The  time  available  to  conduct  exercises  is  certainly 
related  to  other  constraints.  Lack  of  time  may  prevent  the 
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realistic  use  of  personnel  and  equipment  resources.  For 
example,  personnel  and  equipment  mobility  processing  is 
often  unrealistically  simulated  or  modified  to  meet  exercise 
time  schedules. 

Obviously,  proper  funding  is  crucial  to  the  removal 
or  lessening  of  the  previously  mentioned  constraints; 
however,  funding  was  not  the  most  important  factor  affecting 
realism.  This  situation  indicates  that  seme  actions,  apart 
from  fiscal  constraints,  might  be  taken  to  facilitate  more 
realistic  training.  If  the  current  trend  in  reducing 
exercise  budgets  continues,  funding  will  become  an 
increasingly  important  constraint  to  increased  exercise 
realism. 

The .components  of  Effectiveness 

The  analysis  described  in  Chapter  IV  showed  that  eight 
factors  significantly  impact  exercise  effectiveness.  The 
factors,  in  rank  order,  are  a  follows;  exercise  location, 
availability  of  personnel,  number  of  simulations  and 
assumptions,  funding,  scenario  development,  airspace 
limitations,  availability  of  airlift,  and  senior  leadership 
involvement. 

Exercise  location  was  ranked  as  the  most  significant 
factor  in  developing  exercises  effectiveness.  This  result 
may  have  occurred  because  deploying  and  operating  from  a 
location  other  than  the  home  base  eliminates  many  of  the 
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other  constraints  important  to  achieving  effectiveness. 
Operating  from  a  deployed  training  site  forces  participants 
to  function  in  an  unfamiliar  environment.  This  new 
environment  may  require  units  to  adapt  to  a  different 
terrain,  climate,  and  infrastructure.  As  one  survey 
respondent  put  it,  "Deployments  take  away  the  home  court 
advantage . " 

Some  experts  commented  that  the  most  effective  and 
realistic  exercises  involve  deployments  to  an  Area  of 
Responsibility  (AOR)  where  operational  missions  are  being 
conducted.  However,  other  experts  suggested  that  units  can 
gain  important  benefits  by  simply  moving  and  operating  from 
any  site  removed  from  their  home  base.  The  movement  to  an 
off-base  location  reduces  the  number  of  simulations  and 
assumptions  that  are  normally  made  during  an  exercise.  For 
example,  units  can  not  simulate  the  facilities  and  equipment 
required  to  perform  the  exercise  mission  when  operating  from 
a  deployed  location.  Off-base  exercises  have  other 
benefits.  Exercise  participants  will  not  return  to  the 
comfort  of  home  and  family  at  the  end  of  their  "contingency" 
shift.  These  additional  hardships  introduced  by  operating 
away  from  home  build  unit  cohesiveness  and  confidence  and 
prevent  participants  from  developing  unrealistic 
expectations  of  their  combat  operations.  Additionally,  the 
challenges  units  face  in  operating  from  deployed  locations 
help  them  identify  problems  with  equipment  or  procedures. 
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Obviously,  real  world  commitments  and  funding 
constraints  limit  the  number  of  deployments  in  which  units 
can  participate.  When  exercising  at  the  home  location  is 
the  only  viable  alternative,  exercise  planners  should  limit 
the  number  of  simulations  used  during  an  exercise.  When 
possible,  facilities,  communications,  and  working  and  living 
conditions  should  replicate  those  that  would  be  used  in  a 
deployed  location.  Minimizing  simulations  forces 
individuals  to  operate  in  the  unpredictable  and  unfamiliar 
environment  that  is  encountered  in  combat.  Of  course,  all 
assumptions  can  not  be  eliminated  during  exercises,  but 
commanders  should  be  aware  of  how  their  assumptions  and 
simulations  can  impact  exercise  effectiveness.  As  one 
expert  responded,  "You  can't  make  assumptions  during  war." 

Finally,  senior  leadership  involvement  is  important  to 
an  exercise's  ability  to  improve  readiness.  Senior  leaders 
must  emphasize  the  importance  of  realistic  training  and 
ensure  that  other  unit  activities  do  not  reduce  an 
exercise's  priority  or  access  to  resources.  Unfortunately, 
many  commanders  are  evaluated  on  their  unit's  performance 
during  operational  readiness  inspections  (ORIs).  Thus,  unit 
exercises  tend  to  focus  on  the  areas  identified  as  important 
by  OR I  inspection  teams.  Given  the  funding  constraints 
imposed  on  commanders,  they  will  practice  and  train  in 
preparation  for  inspections  rather  than  developing  and 
practicing  other  challenging  scenarios.  Commanders  have 
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little  incentive  to  provide  time  and  resources  for  risky, 
free-wheeling  exercises  in  which  the  possibility  of  failure 
is  high. 

The  turbulence  in  the  world  economic  and  political 
environment  since  the  end  of  the  Cold  War  has  increased  the 
possibility  of  military  taskings  to  unexpected  locations. 

To  adequately  prepare  forces  to  perform  these  new  missions, 
senior  military  leaders  should  encourage  commanders  to 
conduct  challenging  exercises.  Several  procedures  could  be 
used  to  increase  an  exercise's  ability  to  improve  unit 
flexibility  and  adaptability.  For  example,  exercises  can  be 
conducted  that  require  units  to  use  unsophisticated 
communications  or  to  shift  to  different  exercise  facilities. 
One  expert  suggested  that  exercises  would  be  more  effective 
if  commanders  allowed  operations  to  fail  due  to  logistics 
shortfalls  rather  than  manipulating  exercises  to  ensure 
"success."  Incorporation  of  these  practices  into  exercises 
will  better  prepare  the  Air  Force  for  the  challenges  of  the 
future. 

Airspace  limitations  and  the  availability  of  suitable 
flight  training  areas  limit  the  realism  that  flyers  receive 
during  training.  Live  fire  training,  low  level  flying,  and 
other  maneuvers  are  important  elements  in  developing  an 
aircrew's  tactical  repertoire.  These  limitations  must  be 
addressed  when  developing  exercise  plans. 
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The  time  available  to  conduct  an  exercise,  while 
important  in  creating  realism,  has  little  impact  on  an 
exercise's  effectiveness.  In  fact,  the  artif  cial 
compression  of  events  can  increase  exercise  effectiveness. 
This  "time  crunch”  helps  develop  the  stressful  environment 
similar  to  that  experienced  in  an  actual  contingency. 

Advantages  .and -Disadvantages  of  Increasing  Realism 

The  Delphi  experts  agreed  that  more  realistic  training 
exercises  result  in  a  more  effective  fighting  force. 
Realistic  exercises  have  several  advantages.  Units  gain  a 
better  understanding  of  their  capabilities  and  can  identify 
their  weaknesses  in  current  training  practices  and  doctrine. 
Realistic  and  challenging  exercises  encourage  innovative 
decision  raking  and  help  inexperienced  personnel  develop 
leadership  skills.  Most  importantly,  increasing  realism 
improves  vnit  readiness  by  creating  the  environment  that 
will  most  likely  be  encountered  in  war.  Units  develop 
confidence  and  cohesion  as  they  learn  to  adapt  to  new 
situations  and  to  solve  unfamiliar  problems. 

Some  disadvantages  are  also  associated  with  increased 
exercise  lealism.  Increasing  realism  often  requires 
spending  more  money;  a  difficult  proposition  in  this  time  of 
dwindling  defense  budgets.  Increasing  realism  requires 
that  more  time  be  devoted  to  exercise  planning  and  demands 
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more  time  from  senior  leadership.  Also,  if  troops  are 
billeted  away  from  families  to  simulate  realistic 
conditions,  personal  hardships  and  morale  problems  may  be 
experienced.  Unit  morale  and  confidence  may  suffer  if  units 
fail  to  overcome  the  more  difficult  obstacles  created  by 
realistic  exercises.  Finally,  the  likelihood  of  accidents 
increases  with  greater  realism. 

Building  a  higher  degree  of  realism  into  exercises 
requires  the  development  of  accurate  scenarios,  but  this 
effort  is  wasted  unless  the  scenarios  are  updated 
frequently.  Real  world  operations  are  unique,  and  it  is 
impossible  to  build  a  single,  standard  contingency  mold  that 
can  be  used  to  develop  exercises  that  are  effective  for  all 
scenarios.  It  would  be  a  mistake,  for  instance,  to  pattern 
every  exercise  after  the  Desert  Shield/Storm  experience. 
Lessons  learned  are  important,  but  that  war,  like  all  others 
that  preceded  it,  has  already  been  fought  and  will  not 
likely  be  repeated. 

Another  danger  in  increasing  realism  in  exercises  is 
the  tendency  to  suboptimize  certain  areas  at  the  expense  of 
others.  If  an  exercise  realistically  focuses  on  logistics 
shortfalls  at  the  expense  of  operations,  important  lessons 
can  be  learned,  but  a  valuable  training  opportunity  for 
aircrews  may  be  wasted.  This,  again,  supports  the  argument 
for  varied  scenario  development. 
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Conclusion 


This  research  does  not  address  all  issues  involving 
realism  in  Air  Force  exercises.  One  of  the  objectives  of 
this  effort  is  to  generate  more  research  and  dialogue 
concerning  how  the  Air  Force  prepares  for  combat.  Military 
experts  should  stress  that  exercises  should  not  be  planned 
to  maximize  realism  but  should  be  planned  and  conducted  so 
they  are  as  productive  as  possible.  Realism  for  the  sake  of 
realism  is  not  the  goal.  Exercises  should  be  designed  so 
they  are  as  effective  as  possible. 

While  the  factors  discussed  in  this  research  may  be 
too  broad  to  be  readily  understood  and  applied  to  specific 
scenarios,  they  should  form  the  foundation  for  further 
research.  Perhaps  this  limited  discussion  will  encourage 
others  to  analyze  the  significant  issues  involving  exercise 
realism  and  effectiveness.  For  example,  how  important  is 
realism  in  different  types  of  exercises?  Is  it  possible  to 
measure  realism  and  determine  the  optimum  level  for  a 
particular  scenario?  The  analysis  of  these  and  other 
questions  can  improve  our  ability  to  effectively  and 
efficiently  prepare  for  future  conflicts. 
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Appendix  B: _ The  Questionnaires 


The  First  Questionnaire 

The  following  is  the  text  of  the  first  questionnaire: 


EXERCISE  REALISM  QUESTIONNAIRE 
DELPHI  ROUND  1 


Instructions 

1.  Please  fill  out  the  questionnaire  in  the  manner  most 
convenient  for  you  (pen,  pencil,  typed). 

2.  Questions  are  both  multiple  choice  and  open-ended. 
Additional  comments  are  encouraged.  These  comments  may  be 
short  and  simple,  but  more  thorough  answers  are  encouraged. 
Please  return  the  completed  survey  by  mail  (envelope 
enclosed)  or  fax  NLT  1  March  1994.  The  AFIT/LA  fax  number 
is  DSN  986-7988  or  (513)  476-7988.  The  responses  to  all 
questions  will  be  summarized  and  provided  to  you  with  the 
next  round  of  the  survey. 

Thanks  for  your  participation! 


Definition  of  Key  Terms 

A.  Exercise .  Any  activity  involving  the  operation  of 
military  forces  in  an  artificial  or  simulated  hostile 
environment . 

B.  Realism.  The  degree  to  which  a  scenario  or  exercise 
reflects  the  conditions  a  id  environment  that  a  unit  would 
likely  encounter  in  a  contingency  or  combat  situation. 

C.  Cpmbat  Readiness.  The  condition  of  a  unit  being 
prepared  to  perform  their  wartime  mission. 
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1 .  Name 


2.  How  many  years  of  experience  do  you  have  in  duties 
involving  military  exercises? 

a.  2  years  to  6  years 

b.  6  to  10  years 

c.  10  to  12  years 

d.  More  than  12  years 

3.  Have  you  been  involved  in  any  operational  missions  such 
as  JUST  CAUSE  (Panama),  DESERT  SHIELD/STORM  (Saudia  Arabia), 
or  RESTORE  HOPE  (Somalia)? 

a.  Yes  b.  No 

4.  If  you  answered  "yes"  to  question  3,  how  many 
operational  missions  have  you  been  involved  witn? 


5.  Using  the  scale  below,  how  would  you  rate  the  level  of 
realism  in  Air  Force  exercises?  Use  the  additional  space 
below  to  explain  your  answer  or  to  provide  examples. 


a. 

b. 

c . 

d. 

e . 

VERY 

LOW 

NEITHER  LOW 

HIGH 

VERY 

LOW 

NOR  HIGH 

HIGH 

Comments : 


6.  Which  major  commands  have  you  been  assigned  to? 


7.  Of  the  commands  that  you  have  been  assigned  to,  do  you 
think  that  there  is  a  difference  among  these  major  commands 
as  far  as  the  amount  of  realism  in  exercises?  If  so,  which 
commands  conduct  the  roost  realistic  exercises? 

Comments : _ 
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8.  An  exercise  must  be  realistic  tc  be  effective  in 
improving  the  combat  readiness  of  the  personnel  or  units 
participating  in  the  exercise.  Explain  your  answer. 


a . 

b. 

C . 

d. 

e. 

STRONGLY 

DISAGREE 

DISAGREE 

NEITHER  AGREE 
NOR  DISAGREE 

AGREE 

STRONGLY 

AGREE 

Comments : 


9.  List  at  least  3  factors  (such  as  time  available  for 
conducting  an  exercise)  that  you  believe  significantly 
impact  the  level  of  realism  in  an  exercise.  Rank  these 
factors  by  importance  (with  1  being  the  most  important). 

1.  _ 


2. 


3. 

4. 


5. 


6. 


7. 


8. 


10.  From  the  list  of  factors  that  you  listed  in  question  9, 
which  factors  are  most  important  to  an  exercise's 
effectiveness.  In  other  words,  which  factors  are  necessary 
to  ensure  that  an  exercise  improves  combat  readiness. 

Comments : _ _ 


11.  What  are  some  of  the  advantages  of  increasing  realism 
in  exercises? 

Comments : _ 


B-3 


12.  What  are  some  of  the  disadvantages  of  increasing 
realism  in  exercises? 

Comments: _ _ 


13.  Do  you  believe  that  a  point  exists  where  increasing  an 
exercise's  realism  will  provide  little  additional  value  in 
terms  of  improved  training  or  combat  readiness? 

a.  Yes  b.  No 

14.  If  you  answered  yes  to  question  15,  can  you  list 
specific  examples  or  situations  where  increasing  realism  in 
an  exercise  has  little  benefit? 

Comments:  _  _ _ 


This  completes  the  round  one  survey.  The  responses  of  all 
panel  members  will  be  summarized  and  returned  to  you  in  the 
next  survey.  Thanks  for  your  prompt  response! 
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EXERCISE  REALISM  QUESTIONNAIRE 


DELPHI  ROUND  2 


Instructions 

1.  Thanks  for  your  quick  response  on  the  first 
questionnaire.  The  following  questions  are  a  result  of  our 
analysis  of  your  responses  in  round  one.  The  questions  in 
this  second,  and  last,  round  are  similar  to  the  first 
survey,  but  fewer  open-ended  responses  are  required.  When 
applicable,  questions  include  a  summary  of  panel  responses 
submitted  on  the  first  survey. 

2.  Please  return  the  comple'-ed  survey  by  mail  or  fax  NLT  1 
May  1994.  The  AUT/LA  fax  number  is  DSN  986-7988  or  (513) 
476-7988. 


Thanks  for  your  participation! 


Definition  of  Key  Terms 

A.  Exercise.  Any  activity  involving  the  operation  of 
military  forces  in  an  artificial  or  simulated  hostile 
environment. 


B.  Raalisru.  The  degree  to  which  a  scenario  or  exercise 
reflects  the  conditions  and  environment  that  a  unit  would 
lively  encounter  in  a  contingency  or  combat  situe*  on. 


The  condition  of  a  unit  bei.ig 


prepared  to  perform  their  wartime  mission. 


D.  Effective  exercise.  An  exercise  that  improves  the 
leadiness  of  a  unit  or  individual. 
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Name 


1 . 


2.  Questions  #5  from  the  first  survey  asked  you  to  rate  the 
level  of  realism  in  Air  Force  exercises.  The  responses  are 
summarized  as  follows: 


1st  Survey  Responses:  30%  chose  LOW 

40%  chose  NEITHER  HIGH  NOR  LOW 
30%  chose  HIGH 

As  shown  above,  there  was  no  consensus  on  the  current 
level  of  exercise  realism;  most  responses  indicated  that  the 
level  of  realism  in  exercises  is  neither  high  nor  low.  Do 
you  believe  that  this  amount  of  realism  is  adequate  for  Air 
Force  exercises? 


a . 

b. 

C . 

d . 

e . 

STRONGLY 

DISAGREE 

NEITHER  AGREE 

AGREE 

STRONGLY 

DISAGREE 

NOR  DISAGREE 

AGREE 

3.  Question  #11  and  #12  of  t.»e  first  survey  asked  you  to 
list  the  advantages  and  disadvantages  of  increasing  realism 
in  exercises.  The  following  two  lists  are  a  summary  of 
responses.  Use  the  scale  below  the  lists  to  indicate 
whether  you  agree  with  the  majority  of  these  advantages  and 
disadvantages . 

Advantages  q!  increased  realign; 

1.  Improved  unit  readiness  (cohesion,  confidence, 

adaptability) 

2.  Identify  current  weaknesses  in  training/doctrine 

3.  Betu-jr  assessment  of  equipment  capabilities 

4.  Improved  decision-making  and  innovative  thinking 


a . 

b. 

c . 

d. 

e . 

STRONGLY 

DISAGREE 

DISAGREE 

NEITHER  AGREE 
NOR  DISAGREE 

AGREE 

STRONGLY 

AGREE 
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1.  Increased  cost  (longer  exercises/increased  use  of 

resources) 

2.  Increased  safety  mishaps  (loss  of  equipment  and 

personnel ) 

3.  Increased  hardships  on  families 

4.  Disruption  of  peacetime  missions 


a . 

b. 

c. 

d. 

e . 

STRONGLY 

DISAGREE 

NEITHER  AGREE 

AGREE 

STRONGLY 

DISAGREE 

NOR  DISAGREE 

AGREE 

4.  Question  #9  of  the  first  survey  asked  you  to  list  and 
rank  the  factors  that  significantly  impact  the  level  of 
realism  in  an  exercise.  Attachment  1  contains  all  factors 
listed  by  panel  members. 

Using  the  scale  on  attachment  1,  please  indicate  the 
degree  to  which  you  agree  that  each  factor  is  important  to 
exercise  realism. 


5.  Referring  to  question  #4,  our  analysis  of  the  rankings 
for  the  factors  you  submitted  showed  that  the  following 
factors  received  the  highest  overall  ranking:  availability 
of  personnel,  exercise  location,  scenario  development, 
funding,  and  availability  of  airlift  resources. 

Use  the  space  provided  to  reply  to  the  following 
question  for  each  factor: 


If  all  constraints  on  the  use  of  this  resource  were 
removed,  how  would  exercise  realism  improve? 


1.  Availability  of  personnel 
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2.  Exercise  location 


3.  Scenario  development 


4 .  Funding 


5.  Availability  of  airlift 


6.  Question  #10  of  the  first  survey  asked  you  to  rank  the 
factors  that  are  most  important  to  an  exercise's 
effectiveness  or  ability  to  improve  combat  readiness. 
Attachment  2  is  a  list  of  these  factors. 


Use  the  scale  on  attachment  2  to  indicate  the  degree  to 
which  you  agree  that  each  factor  is  important  to  an 
exercise's  effectiveness. 


7.  Oui  analysis  of  the  rankings  for  the  factors  you 
submitted  concerning  the  effectiveness  of  an  exercise  showed 
that  the  following  factors  received  the  highest  overall 
ranking:  exercise  location,  availability  of  personnel, 
number  of  assumptions/  simulations,  funding,  and  scenar  .o 
development . 


Use  the  space  provided  to  reply  to  question  a)  for 
factors  1,  2,  4,  and  5.  Use  question  bi  for  factor  3: 


a)  How  would  the  elimination  of  constraints  on  this 
resource  during  an  exercise  improve  unit  combat 
effectiveness? 
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b)  How  would  the  elimination  of  this  constraint  improve 
unit  combat  readiness? 

1.  Exercise  location 


2.  Availability  of  personnel 


3.  Number  of  assumptions 


4 .  Funding 


5.  Scenario  Development 


This  completes  the 
panel  members  will 
required).  Thankr 


round  two  survey.  The  responses  of  all 
be  summarized  and  returned  to  you  (if 
for  your  prompt  response! 
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Appendix  C; _ Questionnaire  Responses 


The  responses  to  the  open-ended  questions  from  the 
first  survey  are  shown  below.  Several  questions  from  the 
Delphi  questionnaires  were  based  on  a  Likert  scale,  and  the 
experts  did  not  provide  additional  comments. 

Question  #5. _ How  would  rate  the  level  of  realism  in  Air 

Force  exercises? 

Comments .  "Although  I  believe  we  make  a  sincere 
attempt  to  make  exercises  approximate  to  actua'i 
contingencies,  my  experience  has  been  that  it  is  simply  not 
possible  to  insert  the  decision  making  opportunities  and 
time  pressure  found  in  the  real  world.  With  greater  use  of 
computer-based  training  and  exercises,  this  should  change." 

"Exercises  do  provide  for  training  and  understanding  of 
the  process  to  help  in  decision  making  and  flexibility 
needed  during  a  crisis — I  do  believe  operations  receive  more 
training  realism  than  does  logistics  support.  The  level  of 
realism  ’s  constrained  by  resources  and  budget." 

"Short  of  actual  missiles  flying  and  people  dying,  our 
exercises  approach  a  high  degree  of  realism.  All  the  pieces 
are  in  place,  the  biggest  constraints  are  peacetime/local 
restrictions . " 

"Exercises  are  realistic  in  that  they  are  planned  to 
include  and  execute  weapon  systems.  However,  the  opposing 
forces  are  usually  marginal  due  to  budget  limitations." 
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"It  depends  on  the  exercise.  Some  JCS  exercises  have 
been  pretty  realistic  while  many  local  exercises  are  not." 

"A  lot  depends  on  wl  ch  exercise  the  unit  or  individual 
is  involved  in.  Joint  Readiness  Training  exercises  are  by 
far  the  closest  to  what  the  real  Air  Force  is  involved  in. 
Some  exercises  center  around  certain  things  and,  once 
completed,  personnel  go  back  to  their  motels/billeting . " 

"Most  of  my  experience  has  been  in  Special  Operations. 
The  operational  missions  I've  participated  in  were  very  much 
like  our  exercises." 

"Speaking  only  on  the  deployment  portion  of  the 
exercise,  the  nobility  process  does  a  good  job  of  simulating 
ar  actual  deployment.  However,  when  actual  support  aircraft 
are  r.ct  available  for  loading  it  degrades  the  realism 
significantly." 

"Exercise  realism  needs  refinement  and  more  thought  in 
the  initial  planning  stages." 

"Varies  with  the  exercise,  area  available  for  the  FTX, 
and  hest  nation  willingness  to  let  us  operate." 

"Most  exercises  don't  follow  &  plan,  and  they  need  to 
have  a  fly  away.  Otherwise,  squadrons  don't  realize  how 
they  would  operate  or  if  they  have  the  right  equipment  to 
operate  at  a  deployed  location." 

"Although  the  environment  is  realistic,  many  conditions 
are  altered  to  meet  our  training  objectives.  Time  phase 
action  is  the  most  altered." 
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"Realism  is  exercise  dependent.  However,  rarely  will 
you  find  an  exercise  that  can  put  a  high  level  of  realism  in 
both  paperwork  and  missions.  The  packages  and  effort  going 
into  a  "perfect"  exercise  are  too  large  for  units  (still 
operationally  deployed)  to  be  able  to  effectively  manage." 

"Majority  of  exercise  training  is  designed  to  be 
realistic  and  normally  meets  requirements.  Good  planning  is 
the  key  to  having  a  good  exercise.  Budget  constraints  often 
detract  from  accomplishing  all  exercise  goals  and  in  a  lot 
of  instances  do  not  allow  for  a  proper  force  list  or 
scenario . " 

"Combat  is  extremely  difficult  to  exercise.  The 
unknown  variables  of  live  firing  of  weapons  and  the  use  of 
explosives  is  safety  prohibitive.  Support  forces  are  never 
integrated  into  the  overall  air  operation — too  many  support 
functions  are  simulated  (in  the  AF)  during  exercises." 

"Realism  is  the  unpredictable  input!  Exercises  tend  to 
be  carefully  orchestrated  to  meet  a  set  of  objectives  for 
training  or  evaluation.  To  prevent  embarrassing  moments, 
exercise  play  is  often  carefully  controlled  in  the  name  of 
safety. " 

"Funding  constraints,  resource  availability,  and 
smaller  force  lists  result  in  unrealistic  exercises  in  AMC; 
however,  developing  a  plan  and  using  computer  systems, 
procedures,  and  policies  on  a  daily  basis  does  allow 
"chinks"  to  be  found  and  corrected  resulting  in  an  improved 
process . " 
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Comments .  "Probably  PACAF.  Exercise  planners  and 
evaluators  took  a  truly  no-nonsense  approach.  Senior  wing 
leadership  was  deeply  involved  and  ensured  that  small 
details  were  not  overlooked." 

"Exercises  within  AORs  are  more  realistic  and  provide 
more  effective  training?  exercise  realism  within  USAFE, 

PACAF ,  and  CENTAF  is  about  the  same." 

"Yes,  there  is  a  difference.  AFSOC  exercises  are  much 
more  realistic  than  the  AMC  exercises  I  have  participated 
in.  The  key  difference  is  the  level  of  detailed  planning 
and  joint  coordination  between  AFSOC  aircrews  and  their 
customer. " 

"Yes.  I  thought  the  exercises  I  dealt  with  in  MAC  were 
the  most  realistic;  PACAF  exercises  came  a  close  second." 

"Not  a  lot  of  difference  but  some.  AMC  provides 
training  for  all  AFSCs  through  the  Volant  Scorpion/Phoenix 
Ace  program.  Most  other  commands  center  on  security  police 
and  other  main  combat  units.  All  personnel  could  be  on  the 
front  lines  at  any  given  time.  This  happened  several  times 
in  Somalia." 

"Yes,  AFSOC." 

"Yes,  ACC." 
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"Yes,  Acc  does  the  best  job  of  realistic  training. 

Other  commands  do  not  put  forth  the  wing-wide  effort  that 
ACC  does." 

"The  old  TAC  conducted  more  realistic  exercises  because 
they  trained  as  they  were  going  to  fight." 

"AFSOC;  USAFE  is  primarily  fighter  oriented  and 
extremely  limited  because  of  European  airspace 
control/congestion.  MAC  is  close  behind  AFSOC  but  is 
usually  at  the  whim  of  the  Army.  AFSOC  is  better  at 
interoperability,  works  on  a  smaller  scale,  and  often  has  a 
higher  level  support  for  conducting  its  exercises." 

"I  though  communications  command  had  more  realistic 
exercises  than  ACC  because  ACC  had  too  many  simulations  and 
didn't  really  test  capabilities." 

"No,  but  most  of  experience  has  been  in  ACC." 

"Yes,  ACC  is  the  most  realistic." 

"Yes,  AFSOC  has  more  realistic  exercises;  we  have 
larger  admin/ops  support  packages  on  exercises,  which  is 
more  realistic  of  wartime  Ca." 

"AFSOC  planners  tend  to  make  their  exercises  more  joint 
and  the  level  of  planning  is  more  detailed  than  other 
commands.  Special  ops  forces  definitely  have  a  more 
detailed  joint/combined  planning  effort  during  all 
exercises/training . " 

"PACAF  and  USAFE.  PACAF's  JCS  exercises  have  full  base 
play  from  numbered  Air  Force  on  down.  USAFE  used  to  have 
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good  exercises  but  real  world  deployments  have  reduced 
exercises  to  a  minimum." 

"Yes,  AMC  is  involved  in  their  wartime  mission  of 
moving  cargo/passengers.  This  takes  so  much  time  and  effort 
that  realistic  wartime  skills  training  is  often  a  low 
priority.  Europe  had  the  most  realistic  exercises  because 
the  threat  was  so  close." 

Question  #8.  An  exercise  must  be_  rjej*Lis_ti.c__t_o_be_  efjfgc.ti ve¬ 
in  improving  the  combat  readiness  of__thg_p.ersQDn.el  .or- units 
participating  in  the  exercise. _ Explain  vour  answer . 

Comments .  "Disagree.  Packing  up  and  moving,  then 
setting  up  for  battle  is  one  of  the  hardest  things  to  do. 
Personnel  readiness  is  part  of  daily  training  but  unit 
readiness  is  not  practiced  except  in  exercises.  Even  if 
units  never  fire  a  shot,  they've  still  learned  something 
about  mobility  and  logistics." 

"Disagree.  Some  combat  skills  can  be  practiced  and 
improved  in  carefully  controlled  exercises." 

"Strongly  agree.  When  asking  people  to  engage  in  war, 
the  better  prepared  they  are  for  the  actual  situations  they 
will  encounter  in  combat,  the  greater  the  chance  they  will 
survive  and  complete  the  mission." 

"Strongly  agree.  Exercises  need  to  be  designed  to 
train  as  we  fight.  Theater  orientation,  tactics,  and  joint 
working  environments  enhance  the  benefits  we  receive  from  an 
exercise. " 
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"Strongly  agree.  An  exercise  roust  provide  the  same 
level  of  difficulty  and  confusion  that  would  be  experienced 
by  the  fore  «  it  a  real  world  deployed  location." 

"Disagree.  It  depends  on  the  objective  of  the 
exercise;  it  roust  be  defined.  Some  exercises  may  have 
limited  training  objectives." 

"Strongly  agree.  People  roust  know  what  to  expect  when 
deployed  to  a  bare  base  scenario;  exercises  should  duplicate 
those  living  conditions." 

"Agree.  If  exercises  are  not  realistic,  then  nobody 
learns  anything  other  than  packing  a  pallet.  People  need  to 
have  realistic  exercises  so  they  know  what  to  expect  in 
war . " 

"Strongly  agree.  Train  as  you  intend  to  fight." 

"Strongly  agree.  One  problem  of  past  exercises  is 
their  lack  of  realism — people  showing  up  for  deployment  with 
missing  gear,  etc.  A  couple  of  real  deployments  will  teach 
people  to  remember  these  details." 

"Agree.  it  prevents  people  from  feeling  as  if  they  are 
just  going  through  the  motions." 

"Agree.  Realism  tests  required  skills  under  the  proper 
conditions . " 

"Strongly  agree.  Without  the  stress  and  pressure, 
people  don't  know  how  they  will  react  in  the  real  thing." 

"Strongly  agree.  Unfortunately,  people  are  frequently 
hand-picked  to  deploy  but  are  not  the  ones  who  would 
actually  go  in  war." 
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"Agree.  The  scenario  roust  be  realistic  for  personnel 
to  achieve  an  understanding  of  the  planning  and  execution 
process — know  the  chain  of  command,  the  AOR  and  how  units 
are  integrated  within  the  AOR." 

"Strongly  agree.  Air  Force  people  are  smart  and  will 
see  through  a  charade  in  a  flash.  An  exercise  which  does 
noc  stress  them  or  make  them  stretch  and  become  better 
quickly  becomes  harassment.  The  result  is  reduced 
readiness. " 

Question  #11.  What  are  some  of  the  advantages  of  increasing 
realism  in  exercises? 

Comments .  "Realism  puts  stress  on  areas  that  are 
easily  assumed  away  during  an  exercise." 

"Combat  readiness'  The  unit  develops  cohesion  and 
confidence  when  they  overcome  the  unexpected." 

"Increased  realism  provides  a  better  quality  of 
training  by  exposing  troops  to  what  is  expected  during  a 
real  operation." 

"It  identifies  areas  where  more  training  may  be 
required  or  doctrine  may  need  to  be  changed." 

"Limitations  will  be  known  before  war  begins.  Allows 
units  to  practice  self  support  in  all  areas." 

"If  we  practice  properly  then  we  won't  keep  making  the 
same  mistakes  time  and  time  again.  With  the  force  shrinking 
in  size,  I  think  realism  is  needed  more  than  ever." 
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"Gives  units  more  confidence  in  carrying  out  assigned 
missions . " 

"Realism  improves  training  in  command  and  control 
procedures . " 

"Realism  gives  assessment  of  capabilities  and  may  drive 
hardware  procurement." 

"Learning  objectives  can  be  tied  to  a  practical 
situation  and  solution.  Red  Flag  is  an  excellent  example; 
the  payoffs  have  been  tremendous." 

"Helps  people  learn  to  adapt  to  new  situations  and 
improves  readiness." 

"Sparks  tactical  innovation  and  limits  the  number  of 
surprises  when  facing  the  real  thing.  Results  in  higher 
combat  readiness  and  awareness  and  identifies  shortfalls  in 
training  and  equipment." 

Question  IJL 2,.  wfrat-fljcg  some.  g£  tfre  disadveptaags-ffiJE- 
increasing  -realism  in  .exercises? 

Comments ..  "Money  and  more  accidents." 

"Depletes  resources." 

"Cost.  Units  will  be  forced  to  deploy  more  personnel 
and  equipment." 

"Takes  more  time  and  senior  leadership  involvement." 

"Safety  is  always  a  factor  and  cost  becomes  a  factor  at 
some  point." 

"More  mishaps,  injuries,  and  opportunity  for  failure." 
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"Cost,  impact  on  family  life  and  other  peacetime 
missions. " 

"A  great  deal  of  time  and  energy  is  required  to  make  a 
an  exercise  realistic." 

"None.  Only  positive  things  can  happen  when  exercises 
are  as  realistic  as  possible." 

"Longer  down  time  for  avionics  equipment;  families 
would  experience  inconveniences." 

"Under  current  budget  constraints,  it  is  becoming 
extremely  hard  to  have  a  realistic  exercise." 

Question  *1,4. _ Can  you  list  specific  examples  or  situations 

where  increasing  realism  in  an  exercise  has  little  benefit? 

Comments .  "Increasing  an  exercise  force  list  does  not 
always  increase  the  quality  of  training.  Increasing  sortie 
rate  or  number  of  repetitive  events  at  some  point  becomes 
redundant . " 

"One  Joint  Readiness  Training  exercise  is  based  on  the 
first  ten  days  of  deployment;  realistic  but  not  valuable 
training  because  it  does  not  cover  the  full  realm  of  our 
mission . " 

"A  point  exists  where  increasing  realism  has  little 
benefit,  but  it  is  a  variable.  Depends  on  exercise 
scenario,  level  of  participation,  and  a  host  of  other 
factors . " 

"I  feel  that  conducting  an  exercise  over,  say,  4  days 
is  not  realistic.  If  you've  done  detailed  planning  and  have 


C-10 


a  clear  understanding  of  what  you  want  to  accomplish,  then 
it  will  be  realistic  enough  and  beneficial." 

"Totally  swamping  the  staff  with  "missions"  that  are 
not  intended  to  be  executed  results  in  too  much  paperwork 
and  little  beneficial  training." 

"After  loading  14  C-141s,  do  we  really  need  to  load  one 
more?" 

"The  tendency  in  all  exercises  is  to  exercise  for  the 
sake  of  exercising.  Few  exercises  are  actually  evaluated 
and  these  evaluations,  if  any,  are  not  used  to  improve 
readiness.  Only  JCS  is  manned  for  evaluation  and  applying 
lessons  learned.  Most  others  are  just  doing  what  they  did 
last  year." 

"As  you  concentrate  on  certain  skills  or  train  a 
novice,  realism  adds  distractions  and  confusion  making  it 
harder  for  a  trainee  to  focus  and  learn  single  skills. 
Complicated,  realistic  scenarios  are  for  cohesive,  well- 
trained  units." 

"Safety  and  cost  would  be  the  factors  that  determine 
diminishing  returns." 

"Once  a  unit  understands  the  key  elements  in  the 
process  such  as  the  planning,  execution,  integration,  and 
sustainment,  diminishing  returns  apply." 

"Permanent  kill  removal  is  a  good  example.  To  kill  a 
guy  and  send  him  home  before  he  ever  reaches  the  target  area 
will  make  a  point  he  won't  forget  but  the  training  he  has 
lost  by  not  continuing  far  outweighs  the  gain." 
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Appendix  D, _ Analysis  of  Delphi  Questionnaires 

Question  #8  of  the  first  survey  asked  the  experts  to 
identify  the  major  command  that  onducts  the  most  realistic 
exercises.  The  following  describes  the  scaling  technique  used  to 
rate  each  expert's  vote: 

1)  A  factor  was  established  based  on  the  number  of  times  a 
particular  major  command  was  rated  "most  realistic."  The  factors 
were  assigned  as  follows: 


Frequency  Factor 

1  to  2  0.0 
3  to  4  0.5 
5  or  greater  1 . 0 


This  factor  was  multiplied  by  .25  to  obtain  a  frequency 
factor  ( F) . 

2)  The  following  factors  were  assigned  based  on  the  years  cf 
exercise  experience  of  the  expert: 


Frequency  (years) 

Factor 

less  than  6 

0.00 

6  to  10 

0.50 

10  to  12 

0.75 

greater  than  12 

1 . 00 

This  factor  was  multiplied  to  obtain  a  depth  of  experience 
factor  ( El ) . 

3)  The  following  factors  were  assigned  to  each  expert's  vote 
based  on  the  number  of  major  commands  that  expert  had  served  in: 


Number  of  commands  Factor 

less  than  2  0.33 
3  to  4  0.67 
greater  than  5  1.00 


This  factor  was  multiplied  by  0.5  to  obtain  a  breadth  of 
experience  factor  (E2). 
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4)  The  rating  for  each  expert's  vote  was  the  sum  of  the  frequency 
factor,  the  depth  of  experience  factor,  and  the  breadth  of 
experience  factor  (Rating  =  F  +  El  +  E2). 

5)  All  common  major  command  ratings  were  averaged  to  obtain  a 
ranking  with  1.0  representing  the  highest  realism  score  that 
could  be  achieved.  The  calculations  of  the  ratings  for  each 
expert's  response  are  shown  in  Table  15. 

Tables  16  and  17  illustrate  how  the  weighted  ranking  system 
described  in  Chapter  III  was  applied  to  the  factors  submitted  on 
the  first  survey.  Tables  17  and  18  show  the  mean  Likert 
responses  for  the  factors  supplied  to  the  experts  in  the  second 
survey . 
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Table  15 


Ratings  of  Major  Commands 


L£Y£L 

E 

El 

E2 

RATING 

MAJCOM 

Wing (ACC) 

i.0( .25) 

C  .  50 ( .25) 

0.33( .50) 

.540 

ACC 

Wing (ACC) 

1.0( .25) 

0 . 75( .25) 

1.00( .50) 

.938 

ACC 

Wing(AFSOC) 

1 .0( .25) 

0. 50 ( . 25) 

0 . 67 ( .50) 

.710 

AFSOC 

Wing(AMC) 

.5( .25) 

1.00( .25) 

0.67( .50) 

.710 

AMC 

Wing(AMC) 

1.0( .25) 

0 . 75 ( .25) 

1.00(  .50) 

.938 

AFSOC 

Wing(AMC) 

0 . 5 ( .25) 

1.00( .25) 

0 . 67 ( .50) 

.710 

PACAF 

MAJCOM(ACC) 

1.0( .25) 

0 . 75 ( .25) 

0 . 67 ( .50) 

.773 

ACC 

MAJCOM ( ACC ) 

1.0( .25) 

1.00( .25) 

0 . 67 ( .50) 

.835 

ACC 

MAJCOM ( ACC ) 

0.5( .25) 

0.75( . 25) 

0.67( .50) 

.648 

AMC 

MAJCOM (ACC) 

No 

response 

MAJCOM (AFSOC) 

1.0( .25) 

1.0( .25) 

0 . 3  3 ( .50) 

.665 

AFSOC 

MAJCOM (AFSOC) 

1.0( .25) 

1.0( .25) 

0 . 3  3 ( .50) 

.665 

AFSOC 

MAJCOM (AFSOC) 

1 .0( .25) 

1.0( .25) 

0 . 67 ( .50) 

.835 

AFSOC 

MAJCOM (AMC) 

1.0( .25) 

1.00( .25) 

1.00( .50) 

1.000 

ACC 

MAJCOM (AMC) 

0 . 5 ( .25) 

0.75( . 25) 

1.00( .50) 

.813 

AMC 

MAJCOM (AMC 

No  preference 

HQ  USAF 

0 . 5 ( .25) 

1.0( .25) 

1.0( .50) 

.875 

PACAF 

HQ  USAF 

0 . 5 ( .25) 

1.0( .25) 

1.0( .50) 

.875 

PACAF 

HQ  USAF 

0.5( .25) 

!.0( .25) 

1 .0( .50) 

.875 

PACAF 

* 

1.0(  .25) 

1.0( .25) 

1.0( .50) 

1.000 

ACC 

* 

0.0( .25) 

1.0( .25) 

1.0( .50) 

.750 

USAFE 

*  indicates  that  expert  voted  for  more  than  one  command 
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ANALYSIS  OF  FACTORS  AFFECTING  REALISM 
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TABLE  16  (CONTINUED) 
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LIKERT  RESPONSES  FOR  FACTORS  AFFECTING  REALISM 
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TABLE  18  (CONTINUED) 


LIKERT  RESPONSES  FOR  FACTORS  AFFECTING  COMBAT  READINESS 
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12.  Reluctance  to  allow  "operations"  to  fail 
due  to  support  function  shortcomings 
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